The types of errors during speech production can vary across individuals with chronic post-stroke aphasia, possibly due to the location and extent of brain damage. In this study, we evaluated the relationship between semantic vs. phonemic errors during confrontational naming, and their relationship with the degree of damage to ventral and dorsal white matter pathways extending beyond the necrotic stroke lesion. Based on the dual stream model of language processing, we tested the hypothesis that semantic errors would be associated with ventral stream damage, whereas phonemic errors would be associated with dorsal stream damage, but not vice-versa. Multi-shell diffusion MRI was used to obtain kurtosis-based white matter tractography from 32 chronic stroke survivors. Using diffusion microstructural tissue modeling, we estimated axonal loss along the length of the inferior and superior longitudinal fasciculi (ILF and SLF), representing the main pathways in the ventral and dorsal streams, respectively. The frequency of semantic paraphasias was strongly associated with ILF axonal loss, whereas phonemic paraphasias were strongly associated with SLF axonal loss, but not vice versa. This dissociation between semantic and phonological processing is in agreement with the dual stream model of language processing and corroborates the concept that, during speech production, knowledge association (semantics) depends on the integrity of ventral, whereas form encoding (phonological encoding) is more localized to dorsal pathways. These findings also demonstrate the importance of the residual integrity of specific white matter pathways beyond regional gray matter damage for speech production.