Observers are inaccurate when judging the gaze direction of eyes shown in negative rather than positive polarity. On the basis of this polarity effect, it has been proposed that gaze is perceived as directed from the dark part of the eye. Our experiment investigated whether direction judgments simply follow this heuristic, as has been suggested. Participants judged the gaze direction of eyes shown at different eccentricities in positive or negative polarity. The error pattern revealed that most errors were incorrect "straight" judgments, suggesting that judgments do not merely follow the heuristic "the dark part does the looking." We suggest that gaze judgments are based on the outcome of a competition between gaze direction signals: Whereas luminance cues may indicate that gaze is directed from the dark part of the eye, geometric cues may indicate the opposite. This interpretation is supported by reduced overall error rates, and fewer incorrect "straight" responses, for more extreme gaze directions.