The evidence for amygdala processing of emotional items outside the focus of attention is mixed. We hypothesized that differences in attentional demands may, at least in part, explain prior discrepancies. In the present study, attention was manipulated by parametrically varying the difficulty of a central task, allowing us to compare responses evoked by unattended emotion-laden faces while the attentional load of a central task was varied. Reduced responses to unattended emotional stimuli may also reflect an active suppression of amygdala responses during difficult non-emotional tasks (cognitive modulation). To explicitly assess cognitive modulation, an experimental condition was used in which subjects performed the central task without the presence of irrelevant emotional stimuli. Our findings revealed that amygdala responses were modulated by the focus of attention. Stronger responses were evoked during a sex task (when faces were attended) relative to a bar-orientation task (when faces were unattended). Critically, a valence effect was observed in the right amygdala during low attentional demand conditions, but not during medium or high demand conditions. Moreover, performing a difficult non-emotional task alone was associated with signal decreases in a network of brain regions, including the amygdala. Such robust decreases demonstrate that cognitive modulation comprises a powerful factor in determining amygdala responses. Collectively, our findings reveal that both attentional resources and cognitive modulation govern the fate of unattended fearful faces in the amygdala.