Predictive coding of multisensory timing

Abstract

The sense of time is foundational for perception and action, yet it frequently departs significantly from physical time. In the paper we review recent progress on temporal contextual effects, multisensory temporal integration, temporal recalibration, and related computational models. We suggest that subjective time arises from minimizing prediction errors and adaptive recalibration, which can be unified in the framework of predictive coding, a framework rooted in Helmholtz's 'perception as inference'.

Figure I

Schematic illustration of Bayesian inference of duration. The red curve denotes the likelihood P(S|D) for a given duration signal, the blue curve the prior at time n, and the dashed blue curve the updated prior at time n + 1. The dark green curve is the posterior based on Bayesian inference. There are two updating processes: the posterior updating based on the cues and the prior is for reliable sensory estimates, and the prior updating based on error correction is for minimizing forthcoming prediction errors.

Figure 1

Regression to the mean in a time-reproduction task. Upper graphs: average reproduction intervals as a function of physical reproduction for measurements made in three different durations: short (squares, 494–847 ms), intermediate (circles, 671–1024 ms), and long (triangle, 847–1200 ms). The graphs at left refer to control subjects with no musical training, those in center to trained drummers, at right to trained string-instrumentalists. All the non-drummers show a strong regression to the mean of that particular interval: drummers respond veridically. Lower graphs: response distributions for interval 850 ms, taken from the three different conditions (short, intermediate, and long; symbols as before). The direction of the bias depends on the interval, tending to under-estimation for the short interval and overestimation for the long interval. Adapted from [8••].

Figure 2

(a and b) PSEs for judging the duration of auditory stimuli, as a function of the duration of the distractor, for base duration of 100 and 1000 ms. For the base duration of 100 ms, distractors shifted the PSE, in a way consistent with assimilation (short distractors cause underestimation and vice versa), while there is no effect for base duration of 1000 ms. (c and d) Illustration of how the Bayesian prior interacts with the likelihood estimate of duration within the Bayesian model, for 100 ms and 1000 ms base durations. The prior (in green) is assumed to be the same normalized width in the two conditions, corresponding to a Weber fraction of 0.2. The Weber fraction of the likelihood (red curves) vary with duration, broad at short durations, narrow at long durations. The prior dominates in determining the posterior (black curve) at 100 ms base duration, the likelihood dominates at 1000 ms base duration. Adapted from [25].

Figure 3

(a) Illustration of how the MLE model combines two interval-discrimination cues. Cue 1 is based on the difference between the base interval (τ) and the comparison interval (τ + Δ), which we assume follows Weber’s law: σ₁ = w_fτ + c. Cue 2 comes from synchrony/asynchrony categorization: when one interval falls within the simultaneity window and the other does not, this is a strong cue as to which is longer. The probability of this occurring peaks near the boundary of the simultaneity. We assume the discriminability function is normal distributed in logarithmic. According to the MLE model (see Box 1), the discrimination threshold of the combined cues is ${\sigma }_1{\sigma }_2/\sqrt{{\sigma }_1^2+{\sigma }_2^2}.$ (b) Interval discrimination thresholds for subject CL (dots) as a function the base interval, separated for audiovisual, and visual-auditory conditions (adapted from [35]), and the prediction of the MLE model (curves).

Figure 4

How sensory-motor recalibration is affected by precision. (a) Average recalibration as a function of age. (b) Average thresholds as a function of age. (c) The recalibration effect as a function of thresholds: the correlation is strong (R² = 0.25, p < 0.001, slope = −0.9). Adapted from [12••].