Previous PET and MRI studies have indicated that the degree to which pathology translates into clinical symptoms is strongly dependent on sex with women more likely to express pathology as a diagnosis of AD, whereas men are more resistant to clinical symptoms in the face of the same degree of pathology. Here we use DTI to investigate the difference between male and female white matter tracts in healthy older participants (24 women, 16 men) and participants with mild cognitive impairment (21 women, 12 men). Differences between control and MCI participants were found in fractional anisotropy (FA), radial diffusion (DR), axial diffusion (DA) and mean diffusion (MD). A significant main effect of sex was also reported for FA, MD and DR indices, with male control and male MCI participants having significantly more microstructural damage than their female counterparts. There was no sex by diagnosis interaction. Male MCIs also had significantly less normalised grey matter (GM) volume than female MCIs. However, in terms of absolute brain volume, male controls had significantly more brain volume than female controls. Normalised GM and WM volumes were found to decrease significantly with age with no age by sex interaction. Overall, these data suggest that the same degree of cognitive impairment is associated with greater structural damage in men compared with women.