Specific patterns of anatomical symmetry or asymmetry have been associated with sex differences in human brain structure and function. An observer-independent cytoarchitectonic method for the quantification of cell volume densities and areal borders was used to investigate the size and microstructure of primary auditory cortex (Brodmann area 41) in female (n = 14) and male (n = 13) postmortem brains. The total brain volume-adjusted volume of the primary auditory cortex was significantly larger in women than in men bilaterally. Inverse asymmetry towards the right side, as opposed to well-known asymmetries towards the left side, was more frequent in women. Laminar cell volume densities of BA 41 showed no gender effect. The morphometric data confirm (in part) gender differences in the cerebral organization of primary auditory cortex.