To study the response profile of the face-selective N170 component, an adaptation procedure was employed where adaptor and test stimuli were presented in rapid succession. Test stimuli came from 4 different face categories (upright, inverted, and eyeless faces and eyes-only images). The same face stimuli, as well as upright and inverted houses, served as adaptors. Strong N170 amplitude reductions indicative of adaptation were found for all types of face test stimuli preceded by face adaptors relative to house adaptors, demonstrating that at a generic level, the N170 reflects the activation of face-selective neurons by full faces and by face parts. The highly specific pattern of N170 adaptation effects for different combinations of adaptor and test stimulus categories suggests additional distinct contributions of eye-selective neurons and of face-sensitive neurons that are tuned to deviations from canonical stimulus orientations to the N170 component. Results demonstrate that the N170 is generated by multiple neural sources at both early and later stages of configural face processing and that rapid adaptation techniques provide a powerful tool to dissociate these sources.