Evidence for enhanced multisensory facilitation with stimulus relevance: an electrophysiological investigation

Abstract

Currently debate exists relating to the interplay between multisensory processes and bottom-up and top-down influences. However, few studies have looked at neural responses to newly paired audiovisual stimuli that differ in their prescribed relevance. For such newly associated audiovisual stimuli, optimal facilitation of motor actions was observed only when both components of the audiovisual stimuli were targets. Relevant auditory stimuli were found to significantly increase the amplitudes of the event-related potentials at the occipital pole during the first 100 ms post-stimulus onset, though this early integration was not predictive of multisensory facilitation. Activity related to multisensory behavioral facilitation was observed approximately 166 ms post-stimulus, at left central and occipital sites. Furthermore, optimal multisensory facilitation was found to be associated with a latency shift of induced oscillations in the beta range (14-30 Hz) at right hemisphere parietal scalp regions. These findings demonstrate the importance of stimulus relevance to multisensory processing by providing the first evidence that the neural processes underlying multisensory integration are modulated by the relevance of the stimuli being combined. We also provide evidence that such facilitation may be mediated by changes in neural synchronization in occipital and centro-parietal neural populations at early and late stages of neural processing that coincided with stimulus selection, and the preparation and initiation of motor action.

Figure 1. Behavioral measures of multisensory facilitation.

A. Percent (%) error rate (+SD) for unisensory targets (AT and VT) and audiovisual stimuli with target and irrelevant components: ATVI (auditory target and visual irrelevant), AIVT (auditory irrelevant and visual target) and ATVT (audiovisual dual targets). B. Mean motor reaction times (+SEM) for unisensory (AT and VT) and multisensory stimuli (ATVI, AIVT and ATVT). C. Cumulative density functions (CDFs) of motor reaction times (RTs) for AT, VT, ATVI, AIVT, ATVT stimuli, the summed CDFs for unisensory stimuli (AT+VT) and multisensory stimuli with single target components (AIVT+ATVI).

Figure 2. Event-related potentials (ERPs) for audiovisual stimuli with irrelevant components and dual targets.

ERPs for the audiovisual stimuli ATVI (auditory target and visual irrelevant), AIVT (auditory irrelevant and visual target) and ATVT (audiovisual dual targets) at central (C3 Cz, C4 and CPz), occipital (O1, Oz and O2) and parietal (P3, Pz and P4) electrode sites. * depicts post-hoc outcomes following a significant interaction effect for voltage difference where ATVT is significantly different from both ATVI and AIVT. # depicts a significant main effect for stimulus type where ATVT is significantly different from both ATVI and AIVT.

Figure 3. Event-related potential (ERP) components that significantly differ for dual targets.

Plot of Q-values from Tukey post-hocs comparisons following significant one-way ANOVAs for ATVI (auditory relevant target and visual irrelevant), AIVT (auditory irrelevant and visual relevant target) and ATVT (audiovisual dual relevant targets) ERPs at each time sample and channel. Shaded regions depict when the ERP for ATVT was significantly different from both, ATVI and AIVT ERPs for 12 consecutive samples. White regions p>.05, grey regions p<.05 and black regions p<.01.

Figure 4. Global Field Power (GDF) and regions that significantly differ for dual targets.

GDF measures for ATVI (auditory relevant target and visual irrelevant), AIVT (auditory irrelevant and visual relevant target) and ATVT (audiovisual dual relevant targets) stimuli. Gray bars depict time samples where ATVT significantly differ from both ATVI and AIVT for at least 12 consecutive samples (plot of Q-values from Tukey post-hocs comparisons following significant one-way ANOVAs).

Figure 5. Inter-trial variance for audiovisual stimuli at occipital electrode sites.

A. For occipital electrodes O1 and O2, percentage (%) of increase and decrease in inter-trial variance (ITV) in the 8–40 Hz frequency range for ATVI (auditory target and visual irrelevant), AIVT (auditory irrelevant and visual target) and ATVT (audiovisual dual target) stimuli. B. For each stimulus type at O1 and O2, non-significant (p>.01 determined using a bootstrap procedure) changes in inter-trial variability (ITV) from the baseline are occluded using a white mask. C. Mean alpha (8–13 Hz) and beta (14–30 Hz) ITV across time for electrodes O1 and O2. D. Mean amplitude and latency of peak minima (+SEMs) in the alpha and beta frequency range for O1 and O2 electrodes, (* p<.05 for main effect of stimulus type for two-way ANOVA).

Figure 6. Inter-trial variance for audiovisual stimuli at central electrode sites.

A. For the central electrode sites Cz and CPz, percentage (%) of increase and decrease in inter-trial variance (ITV) in the 8–40 Hz frequency range for ATVI (auditory target and visual irrelevant), AIVT (auditory irrelevant and visual target) and ATVT (audiovisual dual target) stimuli. B. For each stimulus type at Cz and CPz, non-significant (p>.01 determined using a bootstrap procedure) changes in inter-trial variance (ITV) from the baseline are occluded using a white mask. C. Mean alpha (8–13 Hz) and beta (14–30 Hz) inter-trial variance (ITV) across time for electrodes Cz and CPz. D. Mean amplitude and latency of peak minima (+SEMs) in the alpha and beta frequency range for Cz and CPz electrodes (* p<.05 for one-way ANOVAs).

Figure 7. Inter-trial variance for audiovisual stimuli at parietal electrode sites.

A. For the parietal electrode sites P3 and P4, percentage (%) of increase and decrease in inter-trial variance (ITV) in the 8–60 Hz frequency range for ATVI (auditory target and visual irrelevant), AIVT (auditory irrelevant and visual target) and ATVT (audiovisual dual target) stimuli at the parietal electrode sites P3 and P4. B. For each stimulus type at P3 and P4, non-significant (p>.01 determined using a bootstrap procedure) changes in ITV from the baseline are occluded using a white mask. C. Mean alpha (8–13 Hz) and beta (14–30 Hz) inter-trial variance (ITV) across time for electrodes P3 and P4. D. Mean amplitude and latency of peak minima (+SEMs) in the alpha and beta frequency range for P3 and P4 electrodes (*, ∧ and # p<.05 for main effect of stimulus type, hemisphere and the interaction between stimulus type and hemisphere, respectively for the two-way ANOVA).

Figure 8. Time-frequency maps of showing areas related to multisensory facilitation.

Time-frequency maps depicting regions where the ATVT stimulus was significantly different from both the ATVI and AIVT stimulus. Significant differences were identified using a bootstrap procedure with alpha set at .01.