Sensitivity to external demands is essential for adaptation to dynamic environments, but comes at the cost of increased risk of adverse outcomes when facing poor environmental conditions. Here, we apply a novel methodology to perform genome-wide association analysis of mean and variance in ten key brain features (accumbens, amygdala, caudate, hippocampus, pallidum, putamen, thalamus, intracranial volume, cortical surface area, and cortical thickness), integrating genetic and neuroanatomical data from a large lifespan sample (n = 25,575 individuals; 8-89 years, mean age 51.9 years). We identify genetic loci associated with phenotypic variability in thalamus volume and cortical thickness. The variance-controlling loci involved genes with a documented role in brain and mental health and were not associated with the mean anatomical volumes. This proof-of-principle of the hypothesis of a genetic regulation of brain volume variability contributes to establishing the genetic basis of phenotypic variance (i.e., heritability), allows identifying different degrees of brain robustness across individuals, and opens new research avenues in the search for mechanisms controlling brain and mental health.