Correlation detection as a general mechanism for multisensory integration

Abstract

The brain efficiently processes multisensory information by selectively combining related signals across the continuous stream of multisensory inputs. To do so, it needs to detect correlation, lag and synchrony across the senses; optimally integrate related information; and dynamically adapt to spatiotemporal conflicts across the senses. Here we show that all these aspects of multisensory perception can be jointly explained by postulating an elementary processing unit akin to the Hassenstein-Reichardt detector-a model originally developed for visual motion perception. This unit, termed the multisensory correlation detector (MCD), integrates related multisensory signals through a set of temporal filters followed by linear combination. Our model can tightly replicate human perception as measured in a series of empirical studies, both novel and previously published. MCDs provide a unified general theory of multisensory processing, which simultaneously explains a wide spectrum of phenomena with a simple, yet physiologically plausible model.

Figure 1. MCD model.

(a) Schematic representation of the model. The MCD integrates multisensory signals (S_V(t), S_A(t)) through a set of low-pass temporal filters followed by linear operations. The MCD model yields two outputs, MCD_Corr(t) (equation 4) and MCD_Lag(t) (equation 5), representing, respectively, the temporal correlation and lag across the input signals. (b) Time-averaged impulse response function of the MCD. The y axis represents the response of the model to visual and auditory impulses as a function of the lag across the senses (see inset). Blue line and axis represent the time-averaged response of the correlation detector (, equation 6), red line and axis represent the time-averaged response of the lag detector (, equation 7). Note how the correlation detector output (blue) peaks at low lags, whereas the output of the lag detector (red) changes sign depending on which modality comes first.

Figure 2. Stimuli, reverse-correlation analyses and results of the psychophysical experiment.

(a) Experimental setup. Participants sat in front of a white fabric disc covering an LED and a speaker. (b) Examples of stimuli used in the experiment (left side), and their cross-correlation (right). Magenta and green lines represent visual (S_V(t)) and auditory stimuli (S_A(t)), respectively. The top row shows an audiovisual stimulus eliciting high responses; the lower two elicit low and high responses, respectively. Cross-correlation of the first stimulus is high at short lags; in the other two it is higher at negative and positive lags, respectively. (c) Reverse-correlation analyses. Stimuli were classified according to participants' responses, that is, ‘light' vs. ‘sound first' in the temporal order judgment task (or ‘same' vs. ‘different causes' in the causality judgment task, not shown). Classification images were calculated by subtracting the average cross-correlation of trials classified as ‘sound first' from the average cross-correlation of trials classified as ‘light first', and smoothing the results using a Gaussian kernel (σ=20 ms, red line, see also f). (d,f) Classification images (solid lines represent data, dashed lines the model). Positive values on the y axis represent positive association to ‘same cause' or ‘sound-first' responses. Predicted classification images are vertically scaled. (e,g) Model output (equations 6, 7) plotted against human responses. Each dot corresponds to 315 responses, 63 per participant. See Supplementary Fig. 2 for plots of individual observers' data. LED, light-emitting diode.

Figure 3. Comparison of our model to previously published psychophysical results.

Dots represent the empirical data, lines the model prediction. (a) Effects of lag and stimulus rate in a correspondence detection task (data from ref. , Experiment 1). (b) Effects of lag and stimulus rate on correspondence detection (data from ref. , Experiment 3). (c) Effects of temporal frequency (rate) and phase shift of periodic stimuli on synchrony detection (data from ref. , Experiment 1). (d) Synchrony detection for a single pulse (data from ref. , Experiment 1) and random sequences of pulses (temporal rate 80 Hz, data from ref. , Experiment 1). (e) Temporal order judgment task (data from ref. , Experiment 2). (f) Synchrony judgment task (data from ref. , Experiment 1).

Figure 4. MCD and optimal cue integration.

(a) A population of spatially tuned MCDs. Each unit receives information from a limited region of visual and auditory space (b) normalization. The output of each unit (MCD_Corr, equation 4) is normalized across units to get a probability distribution of model response over space. (c) MCD and optimal cue integration. Normalized responses of a population of MCDs tuned to a preferred stimulus dimension (for example, space) to visual and auditory stimuli with a spatial offset. The green and magenta lines represent MCD responses to spatially offset unimodal visual and auditory stimuli, respectively. The blue line represents the response of the MCD to the bimodal audiovisual stimuli, while the black dots represent the prediction of Bayes-optimal integration.