Congenital amusia (or tone deafness) is a lifelong disability that prevents otherwise normal-functioning individuals from developing basic musical skills. Behavioural evidence indicates that congenital amusia is due to a severe deficit in pitch processing, but very little is known about the neural correlates of this condition. The objective of the present study was to investigate the structural neural correlates of congenital amusia. To this aim, voxel-based morphometry was used to detect brain anatomical differences in amusic individuals relative to musically intact controls, by analysing T1-weighted magnetic resonance images from two independent samples of subjects. The results were consistent across samples in highlighting a reduction in white matter concentration in the right inferior frontal gyrus of amusic individuals. This anatomical anomaly was correlated with performance on pitch-based musical tasks. The results are consistent with neuroimaging findings implicating right inferior frontal regions in musical pitch encoding and melodic pitch memory. We conceive the present results as a consequence of an impoverished communication in a right-hemisphere-based network involving the inferior frontal cortex and the right auditory cortex. Moreover, the data point to the integrity of white matter tracts in right frontal brain areas as being key in acquiring normal musical competence.