Distinct Neural Mechanisms of Spatial Attention and Expectation Guide Perceptual Inference in a Multisensory World

Abstract

Spatial attention (i.e., task-relevance) and expectation (i.e., signal probability) are two critical top-down mechanisms guiding perceptual inference. Spatial attention prioritizes processing of information at task-relevant locations. Spatial expectations encode the statistical structure of the environment. An unresolved question is how the brain allocates attention and forms expectations in a multisensory environment, where task-relevance and signal probability over space can differ across sensory modalities. We used functional magnetic resonance imaging in human participants (female and male) to investigate whether the brain encodes task-relevance and signal probability over space separately or interactively across sensory modalities. In a novel multisensory paradigm, we manipulated spatial attention and expectation selectively in audition and assessed their effects on behavioral and neural responses to auditory and visual stimuli. Our results show that both auditory and visual stimuli increased activations in a right-lateralized frontoparietal system, when they were presented at locations that were task-irrelevant in audition. Yet, only auditory stimuli increased activations in the medial prefrontal cortex when presented at expected locations and in audiovisual and frontoparietal cortices signaling a prediction error when presented at unexpected locations. This dissociation in multisensory generalization for attention and expectation effects shows that the brain controls attentional resources interactively across the senses but encodes the statistical structure of the environment as spatial expectations independently for each sensory system. Our results demonstrate that spatial attention and expectation engage partly overlapping neural systems via distinct mechanisms to guide perceptual inference in a multisensory world.SIGNIFICANCE STATEMENT In our natural environment the brain is exposed to a constant influx of signals through all our senses. How does the brain allocate attention and form spatial expectations in this multisensory environment? Because observers need to respond to stimuli regardless of their sensory modality, they may allocate attentional resources and encode the probability of events jointly across the senses. This psychophysics and neuroimaging study shows that the brain controls attentional resources interactively across the senses via a frontoparietal system but encodes the statistical structure of the environment independently for each sense in sensory and frontoparietal areas. Thus, spatial attention and expectation engage partly overlapping neural systems via distinct mechanisms to guide perceptual inference in a multisensory world.

Keywords: attention; expectation; fMRI; multisensory; perceptual decisions; space.

Figure 1.

Experimental design, example stimuli of the psychophysics and fMRI experiment and behavioral results of the fMRI experiment. A, The factorial design manipulated: auditory (A) spatial attention (attended hemifield − full pattern, vs unattended hemifield − striped pattern), A spatial expectation (expected hemifield − dark shade, vs unexpected hemifield− light shade) and stimulus modality (auditory modality − orange, vs visual modality − blue). For illustration purposes and analysis, we pooled over stimulus locations (left/right). Presence versus absence of response requirement is indicated by the hand symbol. B, Number of auditory (orange) and visual (blue) trials in the 2 (A attended vs unattended) × 2 (A expected vs unexpected) design. Presence versus absence of response requirement is indicated by the hand symbol. The fraction of the area indicated by the “Response” hand symbol pooled over the two bars of one particular block type (e.g., block type 1) represents the “general response probability” (i.e., the overall probability that a response is required on a particular trial); the general response probability is greater for block type 1 (90%), where attention and expectation are congruent, than block type 2 (60%), where they are incongruent. The fraction of the area indicated by the Response hand symbol for each bar represents the “spatially selective response probability”, i.e., the probability that the observer needs to make a response conditioned on the signal being presented in a particular hemifield; the spatially selective response probability is greater when unattended signals are presented in the unexpected (71.4%) than expected (38.4%) hemifield. C, fMRI runs included 10 blocks of 20 trials alternating with fixation periods. A fixation cross was presented throughout the entire run. The colors indicate as follows: white, fixation period; green or pink, activation period with auditory attention directed to the left (or right) hemifield. On each trial participants were presented with an auditory or visual stimulus (100 ms duration) either in their left or right hemifield. They were instructed to respond as fast and accurately as possible with their right index finger within a response window of 1400 ms. D, Bar plots show response times (across subjects' mean ± SEM) for each of the six conditions with response requirements in the fMRI experiment. The brackets and stars indicate significance of main effects and interactions. **p < 0.01, ***p < 0.001. Audition, orange; vision, blue; attended, full pattern; unattended, striped pattern; expected, dark shade; unexpected, light shade.

Figure 2.

Auditory (A) unattended > attended for auditory and visual stimuli. Activation increases for A unattended > attended stimuli for auditory (AUD; orange, height threshold: p < 0.001, uncorr., extent threshold k > 0 voxels) and visual (VIS; blue, height threshold: p < 0.001, uncorr., extent threshold k > 0 voxels) stimuli (overlap, yellow) are rendered on an inflated canonical brain. The conjunction of A unattended > attended for auditory and visual stimuli is encircled in yellow (height threshold: p < 0.001, uncorrected, extent threshold k > 0 voxels). Activation increases for A unattended > attended that are greater for auditory than visual stimuli (i.e., interaction) are encircled in white (height threshold: p < 0.001, uncorrected, extent threshold k > 0 voxels). Bar plots show the parameter estimates (across participants mean ± SEM, averaged across all voxels in the black encircled cluster) in the (i) right postcentral/intraparietal sulcus, (ii) right superior frontal gyrus, and (iii) right anterior insula that are displayed on axial slices of a mean image created by averaging the subjects' normalized structural images. The bar graphs represent the size of the effect pertaining to BOLD magnitude in non-dimensional units (corresponding to percentage whole-brain mean). Audition, orange; vision, blue; attended, full pattern; unattended, striped pattern; expected, dark shade; unexpected, light shade.

Figure 3.

Auditory (A) expected > unexpected for auditory and visual stimuli. Activation increases for A expected > unexpected auditory stimuli (orange) are rendered on an inflated canonical brain; they are encircled in white if they are significantly greater for auditory than visual stimuli (i.e., interaction). Height threshold of p < 0.001, uncorrected; extent threshold k > 0 voxels. Bar plots show the parameter estimates (across participants mean ± SEM, averaged across all voxels in the black encircled cluster) in the medial prefrontal cortices (i.e., anterior portions of the superior frontal gyri) that are displayed on axial slices of a mean image created by averaging the subjects' normalized structural images; the bar graphs represent the size of the effect in non-dimensional units (corresponding to percentage whole-brain mean). Audition: orange; vision: blue; attended: full pattern; unattended: striped pattern; expected: dark shade; unexpected: light shade.

Figure 4.

Auditory (A) unexpected > expected for auditory and visual stimuli. Activation increases for A unexpected > expected stimuli for auditory stimuli (orange) are rendered on an inflated canonical brain; they are encircled in white if they are significantly greater for auditory than visual stimuli (i.e., interaction). Height threshold of p < 0.001, uncorrected; extent threshold k > 0 voxels. Bar plots show the parameter estimates (across participants mean ± SEM, averaged across all voxels in the black encircled cluster) in (i) bilateral superior temporal gyri and bilateral intraparietal sulci, (ii) bilateral superior frontal gyri, and (iii) bilateral calcarine cortices that are displayed on axial slices of a mean image created by averaging the subjects' normalized structural images. The bar graphs represent the size of the effect in non-dimensional units (corresponding to percentage whole-brain mean). Audition, orange; vision, blue; attended, full pattern; unattended, striped pattern; expected, dark shade; unexpected, light shade.

Figure 5.

Additive effects of auditory (A) attention and expectation in audition (AUD). Activation increases common (i.e., conjunction) for A attention and expectation main effects in the auditory modality ([A unattended > attended AUD] ∩ [A unexpected > expected AUD]) are rendered in orange on an inflated canonical brain; height threshold of p < 0.001, uncorrected; extent threshold k > 0 voxels.