Supervised calibration relies on the multisensory percept

Abstract

Multisensory plasticity enables us to dynamically adapt sensory cues to one another and to the environment. Without external feedback, "unsupervised" multisensory calibration reduces cue conflict in a manner largely independent of cue reliability. But environmental feedback regarding cue accuracy ("supervised") also affects calibration. Here we measured the combined influence of cue accuracy and cue reliability on supervised multisensory calibration, using discrepant visual and vestibular motion stimuli. When the less reliable cue was inaccurate, it alone got calibrated. However, when the more reliable cue was inaccurate, cues were yoked and calibrated together in the same direction. Strikingly, the less reliable cue shifted away from external feedback, becoming less accurate. A computational model in which supervised and unsupervised calibration work in parallel, where the former only relies on the multisensory percept, but the latter can calibrate cues individually, accounts for the observed behavior. In combination, they could ultimately achieve the optimal solution of both external accuracy and internal consistency.

Figure 1. Simulations of multisensory calibration

Two cues for heading direction were generated with a constant heading discrepancy between them (stimulus schematics, leftmost column). One cue was made more reliable (red), and the other, less reliable (blue). For each cue, psychometric plots (cumulative Gaussian functions; second column) were simulated to represent the ratio of rightward choices as a function of heading direction. Each psychometric plot’s intersection with the horizontal dashed line marks its point of subjective equality (PSE, estimate of straight ahead). Corresponding probability distributions (third column) represent the perceived heading for each cue in response to a particular heading stimulus, with biases equal in magnitude, but opposite in sign, to the PSEs. Vertical dotted lines represent accurate perception, according to external feedback. Dark colors represent pre-calibration (baseline) behavior, lighter colors, post-calibration, and the connecting horizontal arrows mark the cues’ shifts. The time-course of calibration is presented in the rightmost column, with the vertical arrows demonstrating complete calibration (corresponding to the post-calibration psychometric and probability distribution plots). (A) During unsupervised calibration, the cues shift towards one-another, achieving internal consistency. For supervised calibration: (B) According to Model 1, only the inaccurate cue is calibrated, both when it is less reliable (top, blue) and more reliable (bottom, red). (C) According to Model 2, only the combined cue is used and thus both cues are calibrated according the combined cue (green).

Figure 2. Examples of supervised visual-vestibular calibration

Four example sessions of supervised calibration are presented (each combination of the cues being more/less reliable and accurate/inaccurate). The heading discrepancy for all examples here was Δ = 10° (vestibular to the right of visual). Plotting conventions are similar to Figure 1, except that red colors represent the visual cue, and blue colors, the vestibular cue. Stimulus schematics (left column) indicate which of the discrepant cues was accurate (lies on vertical dotted line), according to external feedback. Psychometric data (circles, middle column) were fitted with cumulative Gaussian functions, and corresponding probability distributions were generated (right column). Horizontal arrows mark significant shifts of the cues, with the shift size in degrees presented below the arrows on the probability distribution plots. (A) When the more reliable cue was also accurate, only the other (less reliable and inaccurate cue) was calibrated. (B) When accuracy and reliability were dissociated (the less reliable cue was accurate), cue yoking was observed, i.e., both cues shifted together in the same direction. ‘X’ symbols mark shifts of the less reliable cue away from feedback, becoming less accurate.

Figure 3. Population shifts

Population-averaged psychometric functions and PSE histograms are presented for each condition. All data are presented in the form of Δ = +10°, but also comprise data collected with Δ = −10° (the latter were flipped for pooling). Color conventions are the same as Figure 2. Significant shifts are marked by horizontal arrows on the psychometrics, and ‘*’ symbols between the PSE 95% confidence intervals (horizontal bars above the histograms). (A) When the more reliable cue was also accurate, only the other (less reliable and inaccurate cue) was calibrated. (B) When accuracy and reliability were dissociated (the less reliable cue was accurate) cue yoking was observed, i.e., both cues shifted together in the same direction. ‘X’ symbols mark shifts of the less reliable cue away from feedback, becoming less accurate. See also Supplemental Figure S1.

Figure 4. Individual shifts

Visual versus vestibular PSE shifts were plotted in the different conditions. Data for vestibular-accurate sessions are plotted in purple and data for visual-accurate sessions are plotted in orange. An ellipse around each data-point marks the standard error of the mean (SEM) calculated for each cue, individually. Data are presented in the form of Δ = +10°, but also comprise data collected with Δ = −10°. (A) When the more reliable cue was accurate, the data lie on the axes (dotted lines), indicating that only the inaccurate cue was calibrated. The accurate cue did not shift. (B) When the less reliable cue was accurate, the data lie in the quadrants of positive correlation, demonstrating cue yoking. Cue yoking was demonstrated for both monkeys (top plot, B) and humans (bottom plot, B).

Figure 5. Model 3 simulation

Two cues for heading direction were generated with a constant heading discrepancy between them, and calibration was simulated according to Model 3. Pre-calibration, one cue was accurate (blue), and the other, biased (red) in relation to external feedback. Psychometric plots and probability density functions were generated similar to Figure 1. But here, post-calibration plots represent partial calibration, as depicted by the vertical dashed lines and arrows in the time-course plots (rightmost column). Green arrows represent the supervised, yoked, component, and blue and red arrows (for the accurate and inaccurate cue, respectively) represent the unsupervised, individual, component. (A) When the accurate cue was also more reliable, the combined cue bias was small (Δcomb; left column) and hence the yoked component was small. In total this resulted in a shift of the inaccurate but not accurate cue. (B) When accuracy and reliability were dissociated, the combined cue bias was large and hence the yoked component was dominant. This resulted in a shift of the accurate cue together with the inaccurate cue.

Figure 6. Example model fit

Circles mark the average visual and vestibular shifts and error bars represent the SEM (red and blue, respectively; in column 8, the blue circles are partially occluded by the red circles). The data were grouped (from left to right) by low, medium and high visual to vestibular reliability ratios. Each group comprises three columns which depict (from left to right) unsupervised calibration, vestibular accurate (supervised calibration), and visual accurate (supervised calibration). Solid curves represent the model simulations as a function of calibration trials, which were fit simultaneously to all the data at 500 trials (vertical dashed lines). Horizontal dotted lines mark the externally accurate solution. The model fit parameters are presented in column 9. (B) The model fits were assessed by the mean squared error (MSE) between the model simulations and the data, for each condition (red and blue, for visual and vestibular errors respectively). See also Supplemental Figures S2 and S3 for Model 3 and 4 fits for all monkeys.

Figure 7. Model fit comparison

(A) Scatter plots demonstrate the model predicted vs. actual cue shifts (red and blue for visual and vestibular, respectively). The diagonal is marked by a solid black line, and represents a perfect model fit. R² values indicate the data fit to the diagonal. (B) Circles and horizontal bars mark the mean squared error and the Tukey-Kramer 95% confidence intervals, respectively (‘*’ indicates p<0.05; ‘**’ indicates p<0.01).