The Neurophysiological Basis of the Trial-Wise and Cumulative Ventriloquism Aftereffects

Abstract

Our senses often receive conflicting multisensory information, which our brain reconciles by adaptive recalibration. A classic example is the ventriloquism aftereffect, which emerges following both cumulative (long-term) and trial-wise exposure to spatially discrepant multisensory stimuli. Despite the importance of such adaptive mechanisms for interacting with environments that change over multiple timescales, it remains debated whether the ventriloquism aftereffects observed following trial-wise and cumulative exposure arise from the same neurophysiological substrate. We address this question by probing electroencephalography recordings from healthy humans (both sexes) for processes predictive of the aftereffect biases following the exposure to spatially offset audiovisual stimuli. Our results support the hypothesis that discrepant multisensory evidence shapes aftereffects on distinct timescales via common neurophysiological processes reflecting sensory inference and memory in parietal-occipital regions, while the cumulative exposure to consistent discrepancies additionally recruits prefrontal processes. During the subsequent unisensory trial, both trial-wise and cumulative exposure bias the encoding of the acoustic information, but do so distinctly. Our results posit a central role of parietal regions in shaping multisensory spatial recalibration, suggest that frontal regions consolidate the behavioral bias for persistent multisensory discrepancies, but also show that the trial-wise and cumulative exposure bias sound position encoding via distinct neurophysiological processes.SIGNIFICANCE STATEMENT Our brain easily reconciles conflicting multisensory information, such as seeing an actress on screen while hearing her voice over headphones. These adaptive mechanisms exert a persistent influence on the perception of subsequent unisensory stimuli, known as the ventriloquism aftereffect. While this aftereffect emerges following trial-wise or cumulative exposure to multisensory discrepancies, it remained unclear whether both arise from a common neural substrate. We here rephrase this hypothesis using human electroencephalography recordings. Our data suggest that parietal regions involved in multisensory and spatial memory mediate the aftereffect following both trial-wise and cumulative adaptation, but also show that additional and distinct processes are involved in consolidating and implementing the aftereffect following prolonged exposure.

Keywords: audiovisual; electroencephalography; multisensory; recalibration; spatial perception; ventriloquism aftereffect.

Figure 1.

Experiment setup and behavioral data. A, Example sequence of AV and A trials (rare V trials are not shown). The yellow speaker is for illustration only; the sound came from speakers placed behind the screen. The participants submitted their response by moving a mouse cursor to the location where they perceived the sound. The confidence rating was only taken in the A trial. B, ΔVA is the distance between the visual and sound stimuli, each located at one of five horizontal locations. Among the nine possible values of ΔVA, only six were used for efficiency. C, Behavioral results: the ventriloquism effect (left) and ventriloquism aftereffect (right), both median values across participants (n = 19); the shaded areas are 95% confidence intervals around the median. Dots show individual participant's data. D, Temporal progression of biases. Shaded areas indicate 95% hybrid bootstrap confidence intervals around the mean. Black dots denote a significant difference between the LT and ST tested with a cluster-based permutation test (p < 0.01; for details, see Materials and Methods). ve: trial-wise ventriloquism effect. vae: trial-wise ventriloquism aftereffect. LT: long-term paradigm. ST: short-term paradigm.

Figure 2.

Predicting the trial-wise aftereffects based on neurophysiological representations. A, In two separate analyses, we quantified the predictive power of EEG-derived representations of either the multisensory discrepancy or the response in the AV trial to predict the trial-wise vae bias in the A trial, either (1) within a paradigm (thick arrows) or (2) across paradigms (dotted arrows). B, Classifier performance (group-level mean, n = 19) for both paradigms (ST and LT) as cross-validated area under the ROC curve (AUC). C, Neurobehavioral models predicting the trial-wise aftereffect within paradigms based on the EEG-derived cerebral encoding of sensory (ΔVA) or motor (R_AV) variables in the AV trial. Graphs show group-level t-maps of the underlying regression betas. D, Neurobehavioral models predicting the aftereffect across paradigms. Horizontal solid lines denote significance based on cluster-based permutation-based statistics (p < 0.01; see Materials and Methods). E, Time course of regression betas (for LDA_ΔVA) for the same data as in C. Left, Within-paradigm analyses. Right, Cross-paradigm analyses. Solid lines indicate the group-level mean, shaded areas are SEM across participants.

Figure 3.

EEG topographies and source maps for the LDA-ΔVA classifier. A, Group-averaged topographies (forward models) and source maps for the three LT-specific time points derived in Figure 2C, left. B, C, Time point common to both paradigms (Fig. 2C, right; B), and for the peak time point in the cross-paradigm analysis in Figure 2D (C). The data are shown as z score-transformed correlations between single trial source activity and the LDA output (see Materials and Methods). For B and C, the correlations were averaged across paradigms.

Figure 4.

Predicting the neurophysiological representations in the A trial based on the previous stimuli. A, In two separate analyses, we quantified the predictive power of the multisensory discrepancy (or motor response) in the AV trial to predict the trial-wise neurophysiological representations of sound location in the A trial, either (1) within a paradigm (thick arrows) or (2) across paradigms (dotted arrows). B, Classifier performance for sound location (group-level mean, n = 19) for (left) both paradigms (ST and LT) or across paradigms (right) as cross-validated area under the ROC curve (AUC). C, Models reflecting the influence of the multisensory discrepancy (ΔVA) or motor (R_AV) variables in the AV trial on the trial-wise representations of sound location, within each paradigm. Graphs show group-level t-maps of the underlying regression betas. Significance based on cluster-based permutation-based statistics (p < 0.05; see Materials and Methods). D, Same analysis computed across paradigms. E, Time course of regression betas for the results for ΔVA in C and D. Solid lines indicate the group-level mean; shaded areas are SEM values across participants.

Figure 5.

EEG topographies and source maps for the LDA-A_A classifier. A, B, Group-averaged topographies (forward models) and source maps for the LT-specific time point (Fig. 4C, left; A), and for the ST-specific time point (Fig. 4C, right; B). The data are shown as z score-transformed correlations between single-trial source activity and the LDA output (see Materials and Methods). Source maps were averaged across both paradigms given that the LDA forward models were significantly correlated between paradigms.