Exposure to faces biases perceptions of subsequently viewed faces. Faces similar to those seen previously are judged more normal and attractive than they were prior to exposure. Here we show sex-contingent after-effects following adaptation to eye-spacing (experiment 1), facial identity (experiment 2) and masculinity (experiment 3). Viewing faces of one sex with increased eye-spacing and faces of the other sex with decreased eye-spacing simultaneously induced opposite after-effects for male and female faces (assessed by normality judgments). Viewing faces transformed in identity or masculinity increased preferences for novel faces with characteristics similar to those viewed only when the sex of the faces presented in the adaptation phase and in post-adaptation preference tests were congruent. Because after-effects reflect changes in responses of neural populations that code faces, our findings indicate that distinct neural populations code male and female faces.