Calabrian Greeks are an enigmatic population that have preserved and evolved a unique variety of language, Greco, survived in the isolated Aspromonte mountain area of Southern Italy. To understand their genetic ancestry and explore possible effects of geographic and cultural isolation, we genome-wide genotyped a large set of South Italian samples including both communities that still speak Greco nowadays and those that lost the use of this language earlier in time. Comparisons with modern and ancient populations highlighted ancient, long-lasting genetic links with Eastern Mediterranean and Caucasian/Near-Eastern groups as ancestral sources of Southern Italians. Our results suggest that the Aspromonte communities might be interpreted as genetically drifted remnants that departed from such ancient genetic background as a consequence of long-term isolation. Specific patterns of population structuring and higher levels of genetic drift were indeed observed in these populations, reflecting geographic isolation amplified by cultural differences in the groups that still conserve the Greco language. Isolation and drift also affected the current genetic differentiation at specific gene pathways, prompting for future genome-wide association studies aimed at exploring trait-related loci that have drifted up in frequency in these isolated groups.