Accurately reading the body language of others may be vital for navigating the social world, and this ability may be influenced by factors, such as our gender, personality characteristics and neurocognitive processes. This fMRI study examined the brain activation of 26 healthy individuals (14 women and 12 men) while they judged the action performed or the emotion felt by stick figure characters appearing in different postures. In both tasks, participants activated areas associated with visual representation of the body, motion processing and emotion recognition. Behaviorally, participants demonstrated greater ease in judging the physical actions of the characters compared to judging their emotional states, and participants showed more activation in areas associated with emotion processing in the emotion detection task, whereas they showed more activation in visual, spatial and action-related areas in the physical action task. Gender differences emerged in brain responses, such that men showed greater activation than women in the left dorsal premotor cortex in both tasks. Finally, participants higher in self-reported empathy demonstrated greater activation in areas associated with self-referential processing and emotion interpretation. These results suggest that empathy levels and sex of the participant may affect neural responses to emotional body language.