Both transgenderism and homosexuality are facets of human biology, believed to derive from different sexual differentiation of the brain. The two phenomena are, however, fundamentally unalike, despite an increased prevalence of homosexuality among transgender populations. Transgenderism is associated with strong feelings of incongruence between one's physical sex and experienced gender, not reported in homosexual persons. The present study searches to find neural correlates for the respective conditions, using fractional anisotropy (FA) as a measure of white matter connections that has consistently shown sex differences. We compared FA in 40 transgender men (female birth-assigned sex) and 27 transgender women (male birth-assigned sex), with both homosexual (29 male, 30 female) and heterosexual (40 male, 40 female) cisgender controls. Previously reported sex differences in FA were reproduced in cis-heterosexual groups, but were not found among the cis-homosexual groups. After controlling for sexual orientation, the transgender groups showed sex-typical FA-values. The only exception was the right inferior fronto-occipital tract, connecting parietal and frontal brain areas that mediate own body perception. Our findings suggest that the neuroanatomical signature of transgenderism is related to brain areas processing the perception of self and body ownership, whereas homosexuality seems to be associated with less cerebral sexual differentiation.