Most evolutionary explanations for cranial differences between Neandertals and modern humans emphasize adaptation by natural selection. Features of the crania of Neandertals could be adaptations to the glacial climate of Pleistocene Europe or to the high mechanical strains produced by habitually using the front teeth as tools, while those of modern humans could be adaptations for articulate speech production. A few researchers have proposed non-adaptive explanations. These stress that isolation between Neandertal and modern human populations would have lead to cranial diversification by genetic drift (chance changes in the frequencies of alleles at genetic loci contributing to variation in cranial morphology). Here we use a variety of statistical tests founded on explicit predictions from quantitative- and population-genetic theory to show that genetic drift can explain cranial differences between Neandertals and modern humans. These tests are based on thirty-seven standard cranial measurements from a sample of 2524 modern humans from 30 populations and 20 Neandertal fossils. As a further test, we compare our results for modern human cranial measurements with those for a genetic dataset consisting of 377 microsatellites typed for a sample of 1056 modern humans from 52 populations. We conclude that rather than requiring special adaptive accounts, Neandertal and modern human crania may simply represent two outcomes from a vast space of random evolutionary possibilities.