Facial expression and direction of gaze are two important sources of social information, and what message each conveys may ultimately depend on how the respective information interacts in the eye of the perceiver. Direct gaze signals an interaction with the observer but averted gaze amounts to "pointing with the eyes", and in combination with a fearful facial expression may signal the presence of environmental danger. We used fMRI to examine how gaze direction influences brain processing of facial expression of fear. The combination of fearful faces and averted gazes activated areas related to gaze shifting (STS, IPS) and fear-processing (amygdala, hypothalamus, pallidum). Additional modulation of activation was observed in motion detection areas, in premotor areas and in the somatosensory cortex, bilaterally. Our results indicate that the direction of gaze prompts a process whereby the brain combines the meaning of the facial expression with the information provided by gaze direction, and in the process computes the behavioral implications for the observer.