Direct gaze or eye contact is a strong social signal, which triggers distinct processes as compared to other gaze directions. Thus, direct gaze could be early dissociated from averted gaze during the processing of gaze direction. In order to examine the processing of gaze contact relative to averted gaze, we studied evoked potentials in human adults in response to the apparent motion of gaze. Deviated and frontal faces were presented with a fixed gaze direction, followed by an apparent movement of the eyes either toward the subject or away from him/her. The results showed that the perception of direct relative to averted gaze evoked a greater, later and longer lasting N170, suggesting that gaze contact recruited more resources than averted gaze in the early stage of gaze direction processing. Furthermore, direct and averted motion of gaze elicited distinct ERP components between 160 and 210 ms, initiated over centro-parietal electrodes. Source reconstruction revealed the involvement of the Theory-of-Mind network, including the regions of the superior temporal sulcus, the medial prefrontal and the orbitofrontal cortices, in this early dissociation. In addition, the perception of gaze contact relative to averted gaze yielded increased fronto-central P3a and parieto-occipital P3b. All the results were significant whatever the head orientation. Our findings show that gaze contact, as compared to other gaze directions, is an essential social cue which recruits early specific processes. The dissociation between direct and averted gaze processing occurs as soon as 160 ms, involving the social brain network.