The blind subjects have experienced a series of brain structural and functional alterations due to the visual deprivation. It remains unclear as to whether white matter changes differ between blind subjects with visual deprivation before and after a critical developmental period. The present study offered a direct comparison in changes of white matter fractional anisotropy (FA) between congenital blind (CB) and late blind (LB) individuals. Twenty CB, 21 LB (blindness onset after 18 years old), and 40 sight control (SC) subjects were recruited. Both the tract-based spatial statistics (TBSS) and voxel-based analysis (VBA) showed lower FA in the bilateral optic radiations in both blind groups, suggesting that the loss of white matter integrity was the prominent hallmark in the blind people. The LB group showed more extensive white matter impairment than the CB group, indicating the mechanisms of white matter FA changes are different between the CB and LB groups. Using a loose threshold, a trend of an increased FA was found in the bilateral corticospinal tracts in the LB but with a smaller spatial extent relative to the CB. These results suggest that white matter FA changes in the blind subjects are the reflection of multiple mechanisms, including the axonal degeneration, deafferentation, and plasticity.