There is still disagreement among studies with respect to the magnitude, location, and direction of sex differences of local gray matter volume (GMV) in the human brain. Here, we applied a state-of-the-art technique examining GMV in a well-powered sample (n = 2,838) validating effects in two independent general-population cohorts, age range 21-90 years, measured using the same MRI scanner. More GMV in women than in men was prominent in medial and lateral prefrontal areas, the superior temporal sulcus, the posterior insula, and orbitofrontal cortex. In contrast, more GMV in men than in women was detected in subcortical temporal structures, such as the amygdala, hippocampus, temporal pole, fusiform gyrus, visual primary cortex, and motor areas (premotor cortex, putamen, anterior cerebellum). The findings in this large-scale study may clarify previous inconsistencies and contribute to the understanding of sex-specific differences in cognition and behavior.