Motor impairment is common in the elderly population. Disrupted white matter tracts and the resultant loss of connectivity between cortical regions play an essential role in motor control. Using diffusion tensor imaging (DTI), we investigated the effect of white matter microstructure on upper-extremity and lower-extremity motor function in a community-based sample. A total of 766 participants (57.3 ± 9.2 years) completed the assessment of motor performance, including 3-m walking speed, 5-repeat chair-stand time, 10-repeat hand pronation-supination time, and 10-repeat finger-tapping time. Fractional anisotropy (FA), mean diffusivity (MD), and structural network connectivity parameters were calculated based on DTI. Lower FA and higher MD were associated with poor performance in walking, chair-stand, hand pronation-supination, and finger-tapping tests, independent of the presence of lacunes, white matter hyperintensities volume, and brain atrophy. Reduced network density, network strength, and global efficiency related to slower hand pronation-supination and finger-tapping, but not related to walking speed and chair-stand time. Disrupted white matter integrity and reduced cerebral network connectivity were associated with poor motor performance. Diffusion-based methods provide a more in-depth insight into the neural basis of motor dysfunction.