Sight and sound out of synch: fragmentation and renormalisation of audiovisual integration and subjective timing

Abstract

The sight and sound of a person speaking or a ball bouncing may seem simultaneous, but their corresponding neural signals are spread out over time as they arrive at different multisensory brain sites. How subjective timing relates to such neural timing remains a fundamental neuroscientific and philosophical puzzle. A dominant assumption is that temporal coherence is achieved by sensory resynchronisation or recalibration across asynchronous brain events. This assumption is easily confirmed by estimating subjective audiovisual timing for groups of subjects, which is on average similar across different measures and stimuli, and approximately veridical. But few studies have examined normal and pathological individual differences in such measures. Case PH, with lesions in pons and basal ganglia, hears people speak before seeing their lips move. Temporal order judgements (TOJs) confirmed this: voices had to lag lip-movements (by ∼200 msec) to seem synchronous to PH. Curiously, voices had to lead lips (also by ∼200 msec) to maximise the McGurk illusion (a measure of audiovisual speech integration). On average across these measures, PH's timing was therefore still veridical. Age-matched control participants showed similar discrepancies. Indeed, normal individual differences in TOJ and McGurk timing correlated negatively: subjects needing an auditory lag for subjective simultaneity needed an auditory lead for maximal McGurk, and vice versa. This generalised to the Stream-Bounce illusion. Such surprising antagonism seems opposed to good sensory resynchronisation, yet average timing across tasks was still near-veridical. Our findings reveal remarkable disunity of audiovisual timing within and between subjects. To explain this we propose that the timing of audiovisual signals within different brain mechanisms is perceived relative to the average timing across mechanisms. Such renormalisation fully explains the curious antagonistic relationship between disparate timing estimates in PH and healthy participants, and how they can still perceive the timing of external events correctly, on average.

Keywords: Audiovisual integration; Illusions; Individual differences; Psychophysics; Sensory timing.

Fig. 1

T2 weighted images of both lesion sites, outlined in red.

Fig. 2

Trial sequence and stimuli for McGurk (top row) and Stream–Bounce illusions (bottom).

Fig. 3

Psychometric data for PH (with black data points and interpolation using a broken line), and for healthy young (black continuous function) and older (grey) groups. a) TOJ: proportion of ‘voice second’ reports (y-axis) for different auditory lags (negative x-values for auditory lead), interpolated with a logistic function. Horizontals indicate the PSS with 95% confidence intervals based on bootstrapped estimates for PH and on SEMean for controls. b) Phoneme discrimination task: proportion of responses following lip-movement, averaged across incongruous conditions only, interpolated using ADS functions. Auditory lag for tMcG was read off at the maximum.

Fig. 4

PSS (x-axis) plotted against asynchrony for maximum a) McGurk, i.e., tMcG (open circle: PH; grey: older; black: young) and b) bounce illusions (tBounce) with line of best fit, and marginal histograms.

Fig. 5

Temporal renormalisation theory: hypothetical relationship between neural and subjective audiovisual asynchrony. Top left: signals from synchronous auditory and visual stimuli (represented by blue and red disks) converge on different audiovisual mechanisms in the brain via different routes (grey disks). For individual mechanisms the actual stimulus timing cannot be dissociated from the propagation latency. Top right: schematic of the evoked distribution of neural asynchronies, across mechanisms, plotting probability of different asynchronies, as a function of neural asynchrony, with increasing delays of auditory signals relative to visual towards the right. The x-axis text refers to the subjective experience of auditory lead, simultaneity, or auditory lag, given these different neural asynchronies. The neural asynchrony at the central tendency of the distribution is the one which relates most reliably to the objective timing of the auditory and visual stimuli, after delays within individual mechanisms have been averaged out. Following experience with this distribution in natural contexts where objective synchrony is likely, tasks probing mechanisms registering asynchronies near this average may evoke perception of synchrony (marked with a dotted line and ‘Simult’); asynchronies registered within other mechanisms are perceived in proportion to their distance from the average. Lower left: an example where auditory inputs to a subset of mechanisms (towards the right) are particularly delayed. For patient PH it is assumed that these mechanisms contribute to the temporal tuning of the McGurk illusion (labelled McG; see main text), while mechanisms involved in TOJ are preserved. Lower right: the bimodal distribution resulting from delayed auditory input for the McGurk task. The mean of the distribution has shifted towards the auditory-lagged mechanisms serving the McGurk task (labelled McG). The perceived asynchrony within each mechanism is renormalised to this new distribution mean. The result is that neural asynchronies for unaffected mechanisms (here labelled TOJ) originally perceived as synchronous (as in the top example) are now perceived as auditory leading.