Autism spectrum disorder (ASD), attention-deficit/hyperactivity disorder (ADHD), and obsessive-compulsive disorder (OCD) have been associated with difficulties recognizing and responding to social cues. Neuroimaging studies have begun to map the social brain; however, the specific neural substrates contributing to social deficits in neurodevelopmental disorders remain unclear. Three hundred and twelve children underwent structural magnetic resonance imaging of the brain (controls = 32, OCD = 44, ADHD = 77, ASD = 159; mean age = 11). Their social deficits were quantified on the Social Communication Questionnaire (SCQ) and the Reading the Mind in the Eyes Test (RMET). Multivariable regression models were used to examine the structural neuroimaging correlates of social deficits, with both a region of interest and a whole-brain vertex-wise approach. For the region of interest analysis, social brain regions were grouped into three networks: (1) lateral mentalization (e.g., temporal-parietal junction), (2) frontal cognitive (e.g., orbitofrontal cortex), and (3) subcortical affective (e.g., limbic system) regions. Overall, social communication deficits on the SCQ were associated with thinner cortices in the left lateral regions and the right insula, and decreased volume in the ventral striatum, across diagnostic groups (p = 0.006 to <0.0001). Smaller subcortical volumes were associated with more severe social deficits on the SCQ in ASD and ADHD, and less severe deficits in OCD. On the RMET, larger amygdala/hippocampal volumes were associated with fewer deficits across groups. Overall, patterns of associations were similar in ASD and ADHD, supporting a common underlying biology and the blurring of the diagnostic boundaries between these disorders.