Although behavioral studies have suggested that there are gender differences regarding facial recognition, the neural substrates of these differences have not been fully examined. In order to clarify them, we performed a functional magnetic resonance imaging (fMRI) experiment in which participants encoded and recognized male and female faces. Behaviorally, the facial recognition ability of men and women was similar, and was superior for female faces compared to male faces. At the neural level, widespread areas showed greater responses for men vs. women during the encoding and recognition phase, and several areas, including the hippocampal region, showed greater responses to female vs. male faces during recognition. The reduced activation of women's brains during encoding and recognition suggests that the relevant neural systems were more efficiently recruited in women than in men.