Temporal lobe epilepsy with (TLE-mts) and without (TLE-no) mesial temporal sclerosis display different patterns of cortical neuronal loss, suggesting that the distribution of white matter damage may also differ between the sub-groups. The purpose of this study was to examine patterns of white matter damage in TLE-mts and TLE-no and to determine if identified changes are related to neuronal loss at the presumed seizure focus. The 4 T diffusion tensor imaging (DTI) and T1-weighted data were acquired for 22 TLE-mts, 21 TLE-no and 31 healthy controls. Tract-based spatial statistics (TBSS) was used to compare fractional anisotropy (FA) maps and voxel-based morphometry (VBM) was used to identify grey matter (GM) volume atrophy. Correlation analysis was conducted between the FA maps and neuronal loss at the presumed seizure focus. In TLE-mts, reduced FA was identified in the genu, body and splenium of the corpus callosum, bilateral corona radiata, cingulum, external capsule, ipsilateral internal capsule and uncinate fasciculus. In TLE-no, FA decreases were identified in the genu, the body of the corpus callosum and ipsilateral anterior corona radiata. The FA positively correlated with ipsilateral hippocampal volume. Widespread extra-focal GM atrophy was associated with both sub-groups. Despite widespread and extensive GM atrophy displaying different anatomical patterns in both sub-groups, TLE-mts demonstrated more extensive FA abnormalities than TLE-no. The microstructural organization in the corpus callosum was related to hippocampal volume in both patients and healthy subjects demonstrating the association of these distal regions.