Selective attention contributes to perceptual efficiency by modulating cortical activity according to task demands. Visual attention is controlled by activity in posterior parietal and superior frontal cortices, but little is known about the neural basis of attentional control within and between other sensory modalities. We examined human brain activity during attention shifts between vision and audition. Attention shifts from vision to audition caused increased activity in auditory cortex and decreased activity in visual cortex and vice versa, reflecting the effects of attention on sensory representations. Posterior parietal and superior prefrontal cortices exhibited transient increases in activity that were time locked to the initiation of voluntary attention shifts between vision and audition. These findings reveal that the attentional control functions of posterior parietal and superior prefrontal cortices are not limited to the visual domain but also include the control of crossmodal shifts of attention.