Striking variations in brain structure, especially in the gyral patterns of the human cortex, present fundamental challenges in human brain mapping. Probabilistic brain atlases, which encode information on structural and functional variability in large human populations, are powerful research tools with broad applications. Knowledge-based imaging algorithms can also leverage atlased information on anatomic variation. Applications include automated image labeling, pathology detection in individuals or groups, and investigating how regional anatomy is altered in disease, and with age, gender, handedness and other clinical or genetic factors. In this report, we illustrate some of the mathematical challenges involved in constructing population-based brain atlases. A disease-specific atlas is constructed to represent the human brain in Alzheimer's disease (AD). Specialized strategies are developed for population-based averaging of anatomy. Sets of high-dimensional elastic mappings, based on the principles of continuum mechanics, reconfigure the anatomy of a large number of subjects in an anatomic image database. These mappings generate a local encoding of anatomic variability and are used to create a crisp anatomical image template with highly resolved structures in their mean spatial location. Specialized approaches are also developed to average cortical topography. Since cortical patterns are altered in a variety of diseases, gyral pattern matching is used to encode the magnitude and principal directions of local cortical variation. In the resulting cortical templates, subtle features emerge. Regional asymmetries appear that are not apparent in individual anatomies. Population-based maps of cortical variation reveal a mosaic of variability patterns that segregate sharply according to functional specialization and cytoarchitectonic boundaries.