The purpose of this study is to elucidate sex differences in global and regional gray/white matter volume, mean diffusivity (MD), and fractional anisotropy (FA) during normal aging using voxel-based analysis. We studied 245 healthy right-handed subjects with a wide range of ages (115 women, 22-70 years; 130 men, 21-71 years). Regarding global effects, inclusion of a quadratic age term improved the fit to data for white matter fraction and MD, but not for global gray matter volume/fraction or FA. Regarding regional effects, we found anterior-dominant volume loss, FA decrease predominantly in the anterior white matter, and MD increase predominantly in perisylvian regions and periventricular white matter against age for both sexes. Compared with women, we found a steeper FA decline for men in the right inferior fronto-temporal areas, extending to the anterior cingulate cortex, and an accelerated MD increase for men in the bilateral frontal, temporal, and parietal areas. There was no area in which interaction of sex with age was significant for regional volume, or in which a steeper FA decline or accelerated MD increase for women was significant. Our results provide strong evidence of sex dimorphism in global and focal diffusion characteristics during normal aging.