To explore neural correlates of cognitive decline in aging, we used longitudinal behavioral data to identify two groups of older adults (n = 40) that differed with regard to whether their performance on tests of episodic memory remained stable or declined over a decade. Analysis of structural and diffusion tensor imaging (DTI) revealed a heterogeneous set of differences associated with cognitive decline. Manual tracing of hippocampal volume showed significant reduction in those older adults with a declining memory performance as did DTI-measured fractional anisotropy in the anterior corpus callosum. Functional magnetic resonance imaging during incidental episodic encoding revealed increased activation in left prefrontal cortex for both groups and additional right prefrontal activation for the elderly subjects with the greatest decline in memory performance. Moreover, mean DTI measures in the anterior corpus callosum correlated negatively with activation in right prefrontal cortex. These results demonstrate that cognitive decline is associated with differences in the structure as well as function of the aging brain, and suggest that increased activation is either caused by structural disruption or is a compensatory response to such disruption.