The independent effects of different brain pathologies on age-dependent cognitive decline are unclear. We examined this in 300 cognitively unimpaired elderly individuals from the BioFINDER study. Using cognition as outcome we studied the effects of cerebrospinal fluid biomarkers for amyloid-β (Aβ42/40), neuroinflammation (YKL-40), and neurodegeneration and tau pathology (T-tau and P-tau) as well as MRI measures of white-matter lesions, hippocampal volume (HV), and regional cortical thickness. We found that Aβ positivity and HV were independently associated with memory. Results differed depending on age, with memory being associated with HV (but not Aβ) in older participants (73.3-88.4 years), and with Aβ (but not HV) in relatively younger participants (65.2-73.2 years). This indicates that Aβ and atrophy are independent contributors to memory variability in cognitively healthy elderly and that Aβ mainly affects memory in younger elderly individuals. With advancing age, the effect of brain atrophy overshadows the effect of Aβ on memory function.