Top-down control of visual sensory cortex has long been tied to the orienting of visual spatial attention on a rapid, moment-to-moment basis. Here, we examined whether sensory responses in visual cortex are also modulated by natural and comparatively slower fluctuations in whether or not one is paying attention to the task at hand. Participants performed a simple visual discrimination task at fixation as the ERPs to task-irrelevant probes in the upper visual periphery were recorded. At random intervals, participants were stopped and asked to report on their attentional state at the time of stoppage-either "on-task" or "off-task." ERPs to the probes immediately preceding these subjective reports were then examined as a function of whether attention was in an on-task versus off-task state. We found that sensory-evoked responses to the probes were significantly attenuated during off-task relative to on-task states, as measured by the visual P1 ERP component. In two additional experiments, we replicated this effect while (1) finding that off-task sensory attenuation extends to the auditory domain, as measured by the auditory N1 ERP component, and (2) eliminating state-dependent shifts in general arousal as a possible explanation for the effects. Collectively, our findings suggest that sensory gain control in cortex is yoked to the natural ebb and flow in how much attention we pay to the current task over time.