The growth hormone-insulin-like growth factor-1 axis plays a role in normal brain growth but little is known of the effect of growth hormone deficiency on brain structure. Children with isolated growth hormone deficiency (peak growth hormone <6.7 µg/l) and idiopathic short stature (peak growth hormone >10 µg/l) underwent cognitive assessment, diffusion tensor imaging and volumetric magnetic resonance imaging prior to commencing growth hormone treatment. Total brain, corpus callosal, hippocampal, thalamic and basal ganglia volumes were determined using Freesurfer. Fractional anisotropy (a marker of white matter structural integrity) images were aligned and tract-based spatial statistics performed. Fifteen children (mean 8.8 years of age) with isolated growth hormone deficiency [peak growth hormone <6.7 µg/l (mean 3.5 µg/l)] and 14 controls (mean 8.4 years of age) with idiopathic short stature [peak growth hormone >10 µg/l (mean 15 µg/l) and normal growth rate] were recruited. Compared with controls, children with isolated growth hormone deficiency had lower Full-Scale IQ (P < 0.01), Verbal Comprehension Index (P < 0.01), Processing Speed Index (P < 0.05) and Movement-Assessment Battery for Children (P < 0.008) scores. Verbal Comprehension Index scores correlated significantly with insulin-like growth factor-1 (P < 0.03) and insulin-like growth factor binding protein-3 (P < 0.02) standard deviation scores in isolated growth hormone deficiency. The splenium of the corpus callosum, left globus pallidum, thalamus and hippocampus (P < 0.01) were significantly smaller; and corticospinal tract (bilaterally; P < 0.045, P < 0.05) and corpus callosum (P < 0.05) fractional anisotropy were significantly lower in the isolated growth hormone deficiency group. Basal ganglia volumes and bilateral corticospinal tract fractional anisotropy correlated significantly with Movement-Assessment Battery for Children scores, and corpus callosum fractional anisotropy with Full-Scale IQ and Processing Speed Index. In patients with isolated growth hormone deficiency, white matter abnormalities in the corpus callosum and corticospinal tract, and reduced thalamic and globus pallidum volumes relate to deficits in cognitive function and motor performance. Follow-up studies that investigate the course of the structural and cognitive deficits on growth hormone treatment are now required to confirm that growth hormone deficiency impacts significantly on brain structure, cognitive function and motor performance.