Performance on the Sternberg working memory task, and MEG cortical response on a variation of the Sternberg task were examined in middle-aged carriers and non-carriers of the APOE epsilon4 allele. Physical activity was also assessed to examine whether exercise level modifies the relationship between APOE genotype and neurocognitive function. Regression revealed that high physical activity was associated with faster RT in the six- and eight-letter conditions of the Sternberg in epsilon4 carriers, but not in the non-carriers after controlling for age, gender, and education (N=54). Furthermore, the MEG analysis revealed that sedentary epsilon4 carriers exhibited lower right temporal lobe activation on matching probe trials relative to high-active epsilon4 carriers, while physical activity did not distinguish non-carriers (N=23). The M170 peak was identified as a potential marker for pre-clinical decline as epsilon4 carriers exhibited longer M170 latency, and highly physically active participants exhibited greater M170 amplitude to matching probe trials.