The aim of the present voxel-based morphometry study was to examine the link between brain structure and number skills in a group of 6-7-year-old children born prematurely, which are considered to be an at-risk population for mathematical learning disabilities. Therefore, gray and white matter density values were extracted from brain areas previously reported to be relevant for number processing in developing brain systems and, thereafter, correlated with response time results tapping semantic number knowledge [i.e. numerical distance effect (NDE) derived from a number comparison task] as well as with general math proficiency (as indexed by a standardized calculation test). Behavioral results disclosed a significant NDE, thus indicating well-established number magnitude representations for one-digit numerals in our study group. Significant positive correlations between gray matter and NDE emerged in parietal regions (including the right anterior inferior and the left superior parietal lobe) and in the right superior temporal gyrus. Moreover, white matter and NDE were negatively correlated in the right anterior inferior parietal lobe and the right inferior frontal gyrus. Overall, our results are novel insofar as they show that in 6-7-year-old children born prematurely, individual differences in gray and white matter structures are associated with numerical skills. Importantly, in our study group the observed link between brain structure and behavioral performance emerges only regarding an experimental task tapping semantic number knowledge, whereas general math proficiency does not seem to be related to individual differences in brain structure in our study group.