The cross-sex shift hypothesis predicts that gay men should perform more like heterosexual women on important neurocognitive tasks on which men score higher than women, such as mental rotation. Studies also suggest sex differences exist in the neural correlates of mental rotation. However, no studies have taken sexual orientation into account or considered within-group variation attributable to recalled gender nonconformity (a developmental trait reliably associated with human nonheterosexuality). We quantified the neural correlates of mental rotation by comparing two groups of gay men, gender conforming (n = 23) and gender nonconforming (n = 23), to gender conforming heterosexual men (n = 22) and women (n = 22). We observed a sex difference between heterosexual men and women in the premotor cortex/supplementary motor cortex and left medial superior frontal gyrus. We also observed a sex difference as well as a cross-sex shift in gay men who recalled being gender nonconforming as children in the right superior frontal gyrus, right angular gyrus, right amygdala/parahippocampal gyrus, and bilaterally in the middle temporal gyrus and precuneus. Thus, cross-sex shifts may be associated with underlying developmental factors which are associated with sexual orientation (such as gender nonconformity). The results also suggest that gay men should not be studied as a homogenous group.