Previous studies have found that cortical responses to different stimuli become less distinctive as people get older. This age-related dedifferentiation may reflect the broadening of the tuning curves of category-selective neurons (broadening hypothesis) or it may be due to decreased activation of category-selective neurons (attenuation hypothesis). In this study, we evaluated these hypotheses in the context of the face-selective neural network. Over 300 participants, ranging in age from 20 to 89 years, viewed images of faces, houses, and control stimuli in a functional magnetic resonance imaging session. Regions within the core face network and extended face network were identified in individual subjects. Activation in many of these regions became significantly less face-selective with age, confirming previous reports of age-related dedifferentiation. Consistent with the broadening hypothesis, this dedifferentiation in the fusiform face area (FFA) was driven by increased activation to houses. In contrast, dedifferentiation in the extended face network was driven by decreased activation to faces, consistent with the attenuation hypothesis. These results suggest that age-related dedifferentiation reflects distinct processes in different brain areas. More specifically, dedifferentiation in FFA activity may be due to broadening of the tuning curves for face-selective neurons, while dedifferentiation in the extended face network reflects reduced face- or emotion-selective activity.