This study is part of an effort to map neural systems involved in the processing of emotion, and it focuses on the possible cortical components of the process of recognizing facial expressions. We hypothesized that the cortical systems most responsible for the recognition of emotional facial expressions would draw on discrete regions of right higher-order sensory cortices and that the recognition of specific emotions would depend on partially distinct system subsets of such cortical regions. We tested these hypotheses using lesion analysis in 37 subjects with focal brain damage. Subjects were asked to recognize facial expressions of six basic emotions: happiness, surprise, fear, anger, disgust, and sadness. Data were analyzed with a novel technique, based on three-dimensional reconstruction of brain images, in which anatomical description of surface lesions and task performance scores were jointly mapped onto a standard brain-space. We found that all subjects recognized happy expressions normally but that some subjects were impaired in recognizing negative emotions, especially fear and sadness. The cortical surface regions that best correlated with impaired recognition of emotion were in the right inferior parietal cortex and in the right mesial anterior infracalcarine cortex. We did not find impairments in recognizing any emotion in subjects with lesions restricted to the left hemisphere. These data provide evidence for a neural system important to processing facial expressions of some emotions, involving discrete visual and somatosensory cortical sectors in right hemisphere.