Structural brain changes precede cognitive and clinical symptoms in Alzheimer's disease (AD). We aimed to examine the gray and white matter tissue changes in individuals with memory decline over a 12-year period, who might be at risk for AD. The participants were selected from the longitudinal Maastricht Aging Study based on their scores on the verbal word learning task. A group with profound memory decline over a 12-year period (n = 20) was identified and matched with a group that did not meet this criterion (n = 20). All of the participants underwent MRI scanning. Diffusion tensor imaging and cortical thickness analyses were performed to investigate the white and gray matter differences respectively. We found decreased white matter integrity in the memory decline group compared to the control group in frontal and parietal brain regions and in several cortico-cortical and cortico-subcortical tracts. Cortical thinning in the memory decline group was found in frontal, parietal, medial temporal and occipital areas. These results showed similarities with the structural brain changes observed in early AD. Thus, not only may cognitive changes be detected years before the clinical diagnosis, but typical gray and white matter changes appear to be present in older people with memory decline as well. This suggests that a combination of cognitive decline and structural brain changes might be an ideal biomarker for AD pathogenesis.