Behavioral research indicates that healthy aging is accompanied by maintenance of voluntary attentional function in many situations, suggesting older adults are able to use attention to enhance and suppress neural activity. However, other experiments show increased distractibility with age, suggesting a failure of attention. One hypothesis for these apparently conflicting findings is that older adults experience a greater sensory processing load at baseline compared to younger adults. In this situation, older adults might successfully modulate sensory cortical activity relative to a baseline referent condition, but the increased baseline load results in more activity than younger adults after attentional modulation. This hypothesis was tested by comparing average functional brain activity in auditory cortex using quantitative perfusion imaging during resting state and steady-state visual conditions. It was observed that older adults demonstrated greater processing of task-irrelevant auditory background noise than younger adults in both conditions. As expected, auditory activity was attenuated relative to rest during a visually engaging task for both older and younger participants. However, older adults continued to show greater auditory processing than their younger counterparts even after this task modulation. Furthermore, auditory activity during the visual task was predictive of cross-sensory distraction on a behavioral task in older adults. Together, these findings suggest that older adults are more distractible than younger, and the cause of this increased distractibility may lie in baseline brain functioning.