Picture-plane inversion dramatically impairs face recognition. Behavioral and event-related potential (ERP) studies suggest that this effect takes place during perceptual encoding of the face stimulus. However, the relationship between early electrophysiological responses to upright and inverted faces and the behavioral face inversion effect remains unclear. To address this question, we recorded ERPs while presenting 10 subjects with face photographs at 12 different orientations around the clock (30 degrees steps) during an individual face matching task. Using the variability in the electrophysiological responses introduced by the multiple orientations of the target face, we found a correlation between the ERP signal at 130-170 ms on occipito-temporal channels, and the behavioral performance measured on the probe stimulus. Correlations between ERP signal and behavioral performance started about 10 ms earlier in the right hemisphere. Significant effects of orientation were observed already at the level of the visual P1 (peaking at 100 ms), but the ERP signal was not correlated with behavior until the face-sensitive N170 time window. Overall, these observations indicate that plane-inversion affects the perceptual encoding of faces as early as 130 ms in occipito-temporal regions, leading directly to an increase in error rates and RTs during individual face recognition.