Multiple spatial reference frames underpin perceptual recalibration to audio-visual discrepancies

Abstract

In dynamic multisensory environments, the perceptual system corrects for discrepancies arising between modalities. For instance, in the ventriloquism aftereffect (VAE), spatial disparities introduced between visual and auditory stimuli lead to a perceptual recalibration of auditory space. Previous research has shown that the VAE is underpinned by multiple recalibration mechanisms tuned to different timescales, however it remains unclear whether these mechanisms use common or distinct spatial reference frames. Here we asked whether the VAE operates in eye- or head-centred reference frames across a range of adaptation timescales, from a few seconds to a few minutes. We developed a novel paradigm for selectively manipulating the contribution of eye- versus head-centred visual signals to the VAE by manipulating auditory locations relative to either the head orientation or the point of fixation. Consistent with previous research, we found both eye- and head-centred frames contributed to the VAE across all timescales. However, we found no evidence for an interaction between spatial reference frames and adaptation duration. Our results indicate that the VAE is underpinned by multiple spatial reference frames that are similarly leveraged by the underlying time-sensitive mechanisms.

Fig 1. Experiment 1: Illustration of conditions.

In the eye+head-consistent condition (left column), participants maintain central fixation and auditory stimuli are positioned relative to the visual stimulus location: consistent audio-visual spatial disparities are obtained from both eye- and head-centred visual frames. In the eye-consistent condition (middle column), participants maintain fixation at the visual stimulus location, and auditory stimuli are located relative to 0° azimuth: audio-visual spatial disparities remain consistent using eye-centred visual frames, but appear inconsistent using head-centred visual frames. In the head-consistent condition (right column), participants fixate the visual stimulus, and auditory stimuli are positioned relative to the visual stimulus: audio-visual spatial disparities remain consistent using head-centred visual frames, but appear inconsistent using eye-centred visual frames. (a) Stimuli and fixation positions for an example trial pairing a visual stimulus at +30° eccentricity with an auditory stimulus offset -20° leftward. (b) Stimulus locations: head-centred auditory azimuth plotted against visual eccentricity represented in eye-centred (top-row) and head-centred frames (bottom-row). Spatially consistent stimuli should lie parallel to the dotted line. Corner histograms illustrate distributions of audio-visual disparities.

Fig 2. Experiment 1: Group spatial bias and gain estimates.

(a) Spatial bias (intercept) and gain (slope) coefficients for each condition. Error bars indicate standard errors of the coefficients. (b) VAE magnitudes and (c) gain differences, quantified by contrasting spatial bias and gain coefficients between adaptation disparities (-20° > +20°). Positive VAE magnitudes indicate spatial recalibration in the direction of the visual offset. Error bars indicate standard errors of the mean.

Fig 3. Experiment 2: Design, and group spatial bias and gain estimates.

(a) Schematic illustration of an example trial pairing a visual stimulus at +30° eccentricity with an auditory stimulus positioned ±30° of an average -20° leftward offset. (b) Auditory azimuth plotted against visual eccentricity. Eye- and head-centred visual signals yield equivalent audio-visual disparities but which vary over trials. Markers and error bars indicate means and ranges of possible auditory locations for each visual location. Corner histograms illustrate distributions of audio-visual disparities. (c) Spatial bias (intercept) and gain (slope) coefficients for each condition. Error bars indicate standard errors of the coefficients. (d) VAE magnitudes and gain differences, quantified by contrasting spatial bias and gain coefficients between adaptation disparities (-20° > +20°). Values are shown for both constant (blue; Experiment 1: eye+head-consistent condition) and variable disparities (pink; Experiment 2). Positive VAE magnitudes indicate spatial recalibration in the direction of the mean visual offset. Error bars indicate standard errors of the mean.