De novo mutations in specific mTOR pathway genes cause brain overgrowth in the context of intellectual disability (ID). By analyzing 101 mMTOR-related genes in a large ID patient cohort and two independent population cohorts, we show that these genes modulate brain growth in health and disease. We report the mTOR activator gene RHEB as an ID gene that is associated with megalencephaly when mutated. Functional testing of mutant RHEB in vertebrate animal models indicates pathway hyperactivation with a concomitant increase in cell and head size, aberrant neuronal migration, and induction of seizures, concordant with the human phenotype. This study reveals that tight control of brain volume is exerted through a large community of mTOR-related genes. Human brain volume can be altered, by either rare disruptive events causing hyperactivation of the pathway, or through the collective effects of common alleles.