The neuroanatomical correlates of musical structure were investigated using functional magnetic neuroimaging (fMRI) and a unique stimulus manipulation involving scrambled music. The experiment compared brain responses while participants listened to classical music and scrambled versions of that same music. Specifically, the scrambled versions disrupted musical structure while holding low-level musical attributes constant, including the psychoacoustic features of the music such as pitch, loudness, and timbre. Comparing music to its scrambled counterpart, we found focal activation in the pars orbitalis region (Brodmann Area 47) of the left inferior frontal cortex, a region that has been previously closely associated with the processing of linguistic structure in spoken and signed language, and its right hemisphere homologue. We speculate that this particular region of inferior frontal cortex may be more generally responsible for processing fine-structured stimuli that evolve over time, not merely those that are linguistic.