Studies of patients with brain damage, as well as studies with normal subjects have revealed that the right hemisphere is important for recognizing emotions expressed by faces and prosody. It is unclear, however, if the knowledge needed to perform recognition of emotional stimuli is organized by modality or by the type of emotion. Thus, the purpose of this study is to assess these alternative a priori hypotheses. The participants of this study were 30 stroke patients with right hemisphere damage (RHD) and 31 normal controls (NC). Subjects were assessed with the Polish adaptation of the Right Hemisphere Language Battery of Bryan and the Facial Affect Recognition Test based on work of Ekman and Friesen. RHD participants were significantly impaired on both emotional tasks. Whereas on the visual-faces task the RHD subjects recognized happiness better than anger or sadness, the reverse dissociation was found in the auditory-prosody test. These results confirm prior studies demonstrating the role of the right hemisphere in understanding facial and prosodic emotional expressions. These results also suggest that the representations needed to recognize these emotional stimuli are organized by modality (prosodic-echoic and facial-eidetic) and that some modality specific features are more impaired than others.