An individual's brainAGE is the difference between chronological age and age predicted from machine-learning models of brain-imaging data. BrainAGE has been proposed as a biomarker of age-related deterioration of the brain. Having an older brainAGE has been linked to Alzheimer's, dementia, and mortality. However, these findings are largely based on cross-sectional associations which can confuse age differences with cohort differences. To illuminate the validity of brainAGE as a biomarker of accelerated brain aging, a study is needed of a large cohort all born in the same year who nevertheless vary on brainAGE. In the Dunedin Study, a population-representative 1972-73 birth cohort, we measured brainAGE at age 45 years, as well as the pace of biological aging and cognitive decline in longitudinal data from childhood to midlife (N = 869). In this cohort, all chronological age 45 years, brainAGE was measured reliably (ICC = 0.81) and ranged from 24 to 72 years. Those with older midlife brainAGEs tended to have poorer cognitive function in both adulthood and childhood, as well as impaired brain health at age 3. Furthermore, those with older brainAGEs had an accelerated pace of biological aging, older facial appearance, and early signs of cognitive decline from childhood to midlife. These findings help to validate brainAGE as a potential surrogate biomarker for midlife intervention studies that seek to measure dementia-prevention efforts in midlife. However, the findings also caution against the assumption that brainAGE scores represent only age-related deterioration of the brain as they may also index central nervous system variation present since childhood.