Biological differences in male and female sexuality are obvious in the behavioral domain, but the central mechanisms that might explain these behavioral gender differences remain unclear. In this study, we merged two earlier positron emission tomography data sets to enable systematic comparison of the brain responses in heterosexual men and women during sexual tactile genital (penile and clitoral) stimulation and during orgasm. Gender commonalities were most evident during orgasm, a phase which demonstrated activations in the anterior lobe of the cerebellar vermis and deep cerebellar nuclei, and deactivations in the left ventromedial and orbitofrontal cortex in both men and women. During tactile genital stimulation, deactivations in the right amygdala and left fusiform gyrus were found for both genders. Marked gender differences were seen during this phase: left fronto-parietal areas (motor cortices, somatosensory area 2 and posterior parietal cortex) were activated more in women, whereas in men, the right claustrum and ventral occipitotemporal cortex showed larger activation. The only prominent gender difference during orgasm was male-biased activation of the periaqueductal gray matter. From these results, we conclude that during the sexual act, differential brain responses across genders are principally related to the stimulatory (plateau) phase and not to the orgasmic phase itself. These results add to a better understanding of the neural underpinnings of human sexuality, which might benefit treatment of psychosexual disorders.