FMRI investigation of cross-modal interactions in beat perception: audition primes vision, but not vice versa

Abstract

How we measure time and integrate temporal cues from different sensory modalities are fundamental questions in neuroscience. Sensitivity to a "beat" (such as that routinely perceived in music) differs substantially between auditory and visual modalities. Here we examined beat sensitivity in each modality, and examined cross-modal influences, using functional magnetic resonance imaging (fMRI) to characterize brain activity during perception of auditory and visual rhythms. In separate fMRI sessions, participants listened to auditory sequences or watched visual sequences. The order of auditory and visual sequence presentation was counterbalanced so that cross-modal order effects could be investigated. Participants judged whether sequences were speeding up or slowing down, and the pattern of tempo judgments was used to derive a measure of sensitivity to an implied beat. As expected, participants were less sensitive to an implied beat in visual sequences than in auditory sequences. However, visual sequences produced a stronger sense of beat when preceded by auditory sequences with identical temporal structure. Moreover, increases in brain activity were observed in the bilateral putamen for visual sequences preceded by auditory sequences when compared to visual sequences without prior auditory exposure. No such order-dependent differences (behavioral or neural) were found for the auditory sequences. The results provide further evidence for the role of the basal ganglia in internal generation of the beat and suggest that an internal auditory rhythm representation may be activated during visual rhythm perception.

Fig. 1

Tempo judgment paradigm: (A) Illustration of control and test sequences and design summary. Control sequences comprised four elements and test sequences comprised five elements. Sequence elements were marked by tones in the auditory condition and squares in the visual condition. Participants were divided into two groups: half received two sessions in the auditory modality followed by two sessions in the visual modality (AV order), the other half received two sessions of visual followed by two sessions of auditory (VA order). Control and test sequences were randomly intermixed within each session. (B) Task and typical pattern of responses to control sequences and pattern of responses to test sequences when the implied beat is perceived and when the implied beat is not perceived. For control sequences, speeding up or slowing down judgments are made relative to the explicitly heard or seen initial 600-ms interval. For test sequences, judgments can be made relative to the explicitly heard or seen 300-ms initial intervals, or to an implied 600-ms interval. The 600-ms referent is only used when individuals perceive a steady “beat” that is implied by the temporal structure of the test sequences (Grahn and McAuley, 2009).

Fig. 2

Average values of the beat sensitivity index, w, for AV and VA orders for auditory and visual modalities; w values are shown separately for control (Panel A) and test (Panel B) sequences. For control sequences, w values are not affected by order. For test sequences, there is a significant interaction between modality and order, caused by an increase in w for the visual condition in the AV order relative to the VA order. Error bars represent standard error.

Fig. 3

Areas that were significantly active in the conjunction of auditory-rest and visual-rest, pFDR < 0.05, whole-brain corrected.

Fig. 4

Brain areas with significantly greater activity during the visual condition in the auditory-first order compared to the visual-first order (auditory-first visual condition–visual-first visual condition) at pFDR < 0.05, whole-brain corrected. The image is masked by the F-contrast of the interaction between modality and order at p < 0.001 uncorrected to exclude any non-specific order-related differences that were present in both modalities; y and z indicate position in stereotaxic coordinates.

Fig. 5

Mean signal intensities extracted for the brain areas showing significantly greater activity during the visual condition in the auditory-first order compared to the visual-first order.

Fig. 6

A scatterplot showing average bilateral putamen activity and w values from the visual session during fMRI.