Mild cognitive impairment (MCI) is associated with early memory loss, Alzheimer's disease (AD) neuropathology, inefficient or ineffective neural processing, and increased risk for AD. Unfortunately, treatments aimed at improving clinical symptoms or markers of brain function generally have been of limited value. Physical exercise is often recommended for people diagnosed with MCI, primarily because of its widely reported cognitive benefits in healthy older adults. However, it is unknown if exercise actually benefits brain function during memory retrieval in MCI. Here, we examined the effects of exercise training on semantic memory activation during functional magnetic resonance imaging (fMRI). Seventeen MCI participants and 18 cognitively intact controls, similar in sex, age, education, genetic risk, and medication use, volunteered for a 12-week exercise intervention consisting of supervised treadmill walking at a moderate intensity. Both MCI and control participants significantly increased their cardiorespiratory fitness by approximately 10% on a treadmill exercise test. Before and after the exercise intervention, participants completed an fMRI famous name discrimination task and a neuropsychological battery, Performance on Trial 1 of a list-learning task significantly improved in the MCI participants. Eleven brain regions activated during the semantic memory task showed a significant decrease in activation intensity following the intervention that was similar between groups (p-values ranged 0.048 to 0.0001). These findings suggest exercise may improve neural efficiency during semantic memory retrieval in MCI and cognitively intact older adults, and may lead to improvement in cognitive function. Clinical trials are needed to determine if exercise is effective to delay conversion to AD.