We investigated whether face-specific processes as indicated by the N170 in event-related brain potentials (ERPs) are modulated by emotional significance in facial expressions. Results yielded that emotional modulations over temporo-occipital electrodes typically used to measure the N170 were less pronounced when ERPs were referred to mastoids than when average reference was applied. This offers a potential explanation as to why the literature has so far yielded conflicting evidence regarding effects of emotional facial expressions on the N170. However, spatial distributions of the N170 and emotion effects across the scalp were distinguishable for the same time point, suggesting different neural sources for the N170 and emotion processing. We conclude that the N170 component itself is unaffected by emotional facial expressions, with overlapping activity from the emotion-sensitive early posterior negativity accounting for amplitude modulations over typical N170 electrodes. Our findings are consistent with traditional models of face processing assuming face and emotion encoding to be parallel and independent processes.