Memory problems are among the most common complaints as people grow older. Using structural equation modeling of commensurate scores of anterograde memory from a large (N = 315), population-derived sample (www.cam-can.org), we provide evidence for three memory factors that are supported by distinct brain regions and show differential sensitivity to age. Associative memory and item memory are dramatically affected by age, even after adjusting for education level and fluid intelligence, whereas visual priming is not. Associative memory and item memory are differentially affected by emotional valence, and the age-related decline in associative memory is faster for negative than for positive or neutral stimuli. Gray-matter volume in the hippocampus, parahippocampus and fusiform cortex, and a white-matter index for the fornix, uncinate fasciculus and inferior longitudinal fasciculus, show differential contributions to the three memory factors. Together, these data demonstrate the extent to which differential ageing of the brain leads to differential patterns of memory loss.