Alzheimer's disease (AD) is a progressive neurodegenerative disease involving the decline of memory and other cognitive functions. Mild cognitive impairment (MCI) represents a transition phase between normal aging and early AD. The degeneration patterns of the white matter across the brain in AD and MCI remain largely unclear. Here we used diffusion tensor imaging and tract-based spatial statistics (TBSS) to investigate white matter changes in multiple diffusion indices (e.g., fractional anisotropy, axial, radial and mean diffusivities) in both AD and MCI patients. Compared with the normal controls, the AD patients had reduced fractional anisotropy and increased axial, radial and mean diffusivities in widespread white matter structures, including the corpus callosum and the white matter of lateral temporal cortex, the posterior cingulate cortex/precuneus and the fronto-parietal regions. Similar white matter regions with reduced anisotropy were also found in MCI patients but with a much less extent than in AD. Between the AD and MCI groups, there were significant differences in the axial and mean diffusivities of the white matter tracts adjacent to the posterior cingulate cortex/precuneus without anisotropy changes. Taken together, our findings based upon multiple diffusion indices (FA, axial, radial and mean diffusivities) suggest distinct degeneration behaviors of the white matter in AD and MCI.