Episodic memory impairment is a consistent, pronounced deficit in pre-clinical stages of late-onset Alzheimer's disease (AD). Individuals with risk factors for AD exhibit altered brain function several decades prior to the onset of AD-related symptoms. In the current event-related fMRI study of spatial context memory we tested the hypothesis that middle-aged adults (MA; 40-58 yrs) with a family history of late onset AD (MA+ FH), or a combined + FH and apolipoprotein E ε4 allele risk factors for AD (MA+ FH + APOE4), will exhibit differences in encoding and retrieval-related brain activity, compared to - FH - APOE4 MA controls. We also hypothesized that the two at-risk MA groups will exhibit distinct patterns of correlation between brain activity and memory performance, compared to controls. To test these hypotheses we conducted multivariate task, and behavior, partial least squares analysis of fMRI data obtained during successful context encoding and retrieval. Our results indicate that even though there were no significant group differences in context memory performance, there were significant differences in brain activity and brain-behavior correlations involving the hippocampus, inferior parietal cortex, cingulate, and precuneus cortex in MA with AD risk factors, compared to controls. In addition, we observed that brain activity and brain-behavior correlations in anterior-medial PFC and in ventral visual cortex differentiated the two MA risk groups from each other, and from MAcontrols. Our results indicate that functional differences in episodic memory-related regions are present by early midlife in adults with + FH and + APOE-4 risk factors for late onset AD, compared to middle-aged controls.