Bilateral damage to the human amygdala impairs recognition of negatively valenced emotions from facial expressions, but it is unclear if this finding generalizes to richer visual stimuli that contain cues in addition to faces. We investigated this issue in 4 subjects with bilateral amygdala damage, 23 with unilateral amygdala damage, 22 brain-damaged controls and 16 normal individuals. Subjects were shown two blocks of complex social scenes; all stimuli in the two blocks were identical, except that the first block had all facial expressions in the image erased. While control subjects were more accurate in recognizing emotions when facial expressions were present, subjects with bilateral amygdala damage did not show the same benefit for negative emotions, often performing equivalently across the two conditions. Most striking, subjects with bilateral amygdala damage were more accurate in recognizing scenes showing anger with faces erased than with faces present, an effect resulting in part from highly abnormal recognition of certain angry facial expressions. All four subjects with bilateral amygdala damage were impaired in recognizing angry faces shown in isolation, and frequently mistook expressions of anger for smiles, a mistake never made by any control subject. Bilateral amygdala damage thus disproportionately impairs recognition of certain emotions from complex visual stimuli when subjects utilize information from facial expressions.