Aging leads to gray and white matter decline but their causation remains unclear. We explored two classes of models of age and dementia risk related brain changes. The first class of models emphasises the importance of gray matter: age and risk-related processes cause neurodegeneration and this causes damage in associated white matter tracts. The second class of models reverses the direction of causation: aging and risk factors cause white matter damage and this leads to gray matter damage. We compared these models with linear mediation analysis and quantitative MRI indices (from diffusion, quantitative magnetization transfer and relaxometry imaging) of tissue properties in two limbic structures implicated in age-related memory decline: the hippocampus and the fornix in 166 asymptomatic individuals (aged 38-71 years). Aging was associated with apparent glia but not neurite density damage in the fornix and the hippocampus. Mediation analysis supported white matter damage causing gray matter decline; controlling for fornix glia damage, the correlations between age and hippocampal damage disappear, but not vice versa. Fornix and hippocampal differences were both associated with reductions in episodic memory performance. These results suggest that fornix white matter glia damage may cause hippocampal gray matter damage during age-dependent limbic decline.