Cerebral white matter undergoes various changes with normal aging. This study investigated the association between age, gender, and the global and regional fractional anisotropy (FA) and mean diffusivity (MD) in 145 adults (30 to 80 years old) using diffusion tensor magnetic resonance imaging. We studied sixteen regions of interest in both hemispheres to search for regions that display age- and gender-related white matter changes and also performed a complementary voxel-based analysis without any hypothesis a priori. On a global scale, our results indicate that the full brain FA was negatively correlated with age. The regional analysis showed that the anterior corpus callosum, the bilateral anterior and posterior internal capsule, and the posterior periventricular regions had the most significant age-related FA decrease. On the other hand, the FA in the temporal and occipital regions was not correlated with age. However, in contrast to males, females overall had a significantly lower FA in the right deep temporal regions. More gender differences in precentral, cingulate, and anterior temporal white matter areas were also found, suggesting that microstructural white matter organization in these regions may have a sexual dimorphism. Such differences were mainly due to the increase in diffusion perpendicular to fiber tracts.