Longitudinal neuroimaging investigation of fragile X syndrome (FXS), the most common cause of inherited intellectual disability and autism, provides an opportunity to study the influence of a specific genetic factor on neurodevelopment in the living human brain. We examined voxel-wise gray and white matter volumes (GMV, WMV) over a 2-year period in 1- to 3-year-old boys with FXS (n = 41) and compared these findings to age- and developmentally matched controls (n = 28). We found enlarged GMV in the caudate, thalamus, and fusiform gyri and reduced GMV in the cerebellar vermis in FXS at both timepoints, suggesting early, possibly prenatal, genetically mediated alterations in neurodevelopment. In contrast, regions in which initial GMV was similar, followed by an altered growth trajectory leading to increased size in FXS, such as the orbital gyri, basal forebrain, and thalamus, suggests delayed or otherwise disrupted synaptic pruning occurring postnatally. WMV of striatal-prefrontal regions was greater in FXS compared with controls, and group differences became more exaggerated over time, indicating the possibility that such WM abnormalities are the result of primary FMRP-deficiency-related axonal pathology, as opposed to secondary connectional dysregulation between morphologically atypical brain structures. Our results indicate that structural abnormalities of different brain regions in FXS evolve differently over time reflecting time-dependent effects of FMRP deficiency and provide insight into their neuropathologic underpinnings. The creation of an early and accurate human brain phenotype for FXS in humans will significantly improve our capability to detect whether new disease-specific treatments can "rescue" the FXS phenotype in affected individuals.