Many studies have described the diversity of Austronesian-speaking Taiwanese people to shed more light on their origin and their connection with the “Out of Taiwan” migrations. However, the genetic relationship between the non-Austronesian-speaking groups of Taiwan and the populations of continental Asia is still unclear. Here, we studied the diversity of mtDNA in 767 non-Austronesian speakers from 16 locations in Taiwan using partial sequencing obtained from the hypervariable segment I (HVS-I) and coding regions 8,001-9,000 and 9.801–10,900 and 85 complete mtDNA genome sequences. Bayesian analysis of population structure was used to examine their relationship with over 3662 individuals representing indigenous groups of Taiwan, continental East Asia, Japan, and Island Southeast Asia. The whole analysis identified 278 haplotypes. Complete genomes revealed 62 novel subhaplogroups, of which 31 were exclusive to Taiwan. Estimates of coalescence times of all subhaplogroups showed peaks of diversification greater than 5.0 kya, likely characterizing gene flow from continental East Asian groups but not excluding in situ Taiwanese ancestry. Furthermore, a significant number of clades exclusive to non-Austronesian speakers of Taiwan (NAN_Tw) showed coalescence peaks between 1.0 and 2.6 kya, suggesting possible late Neolithic to early metal age settlements of NAN_Tw and local expansion in Taiwan.