Interactions between space and effectiveness in human multisensory performance

Abstract

Several stimulus factors are important in multisensory integration, including the spatial and temporal relationships of the paired stimuli as well as their effectiveness. Changes in these factors have been shown to dramatically change the nature and magnitude of multisensory interactions. Typically, these factors are considered in isolation, although there is a growing appreciation for the fact that they are likely to be strongly interrelated. Here, we examined interactions between two of these factors - spatial location and effectiveness - in dictating performance in the localization of an audiovisual target. A psychophysical experiment was conducted in which participants reported the perceived location of visual flashes and auditory noise bursts presented alone and in combination. Stimuli were presented at four spatial locations relative to fixation (0°, 30°, 60°, 90°) and at two intensity levels (high, low). Multisensory combinations were always spatially coincident and of the matching intensity (high-high or low-low). In responding to visual stimuli alone, localization accuracy decreased and response times (RTs) increased as stimuli were presented at more eccentric locations. In responding to auditory stimuli, performance was poorest at the 30° and 60° locations. For both visual and auditory stimuli, accuracy was greater and RTs were faster for more intense stimuli. For responses to visual-auditory stimulus combinations, performance enhancements were found at locations in which the unisensory performance was lowest, results concordant with the concept of inverse effectiveness. RTs for these multisensory presentations frequently violated race-model predictions, implying integration of these inputs, and a significant location-by-intensity interaction was observed. Performance gains under multisensory conditions were larger as stimuli were positioned at more peripheral locations, and this increase was most pronounced for the low-intensity conditions. These results provide strong support that the effects of stimulus location and effectiveness on multisensory integration are interdependent, with both contributing to the overall effectiveness of the stimuli in driving the resultant multisensory response.

Keywords: Localization; Multisensory; Psychophysics; Race model.

Figure 1. Stimulus Apparatus and Trial Structure

(A, B) Auditory and visual stimuli were presented on a two-monitor array such that stimuli locations at 0°, 30°, 60°, and 90° were equidistant from the nasium. (C) Trial Structure for the task. After fixating a cross for 500 – 1000 ms, subjects were presented with auditory, visual or multisensory stimulus. The subjects then had 2000 ms to respond with the location of the stimulus or indicate that there was no stimulus.

Figure 2. Accuracy Performance Across Space and Intensity

(A, B) Localization performance declined for auditory and visual stimuli for high (A) and low (B) intensity conditions. (C) To assess multisensory enhancement, the greatest unisensory performance was subtracted from the multisensory performance. Enhancements across space follow the pattern of inverse effectiveness.

Figure 3. Mean Response Times Across Space and Intensity

(A, B) Mean RTs generally declined with more peripheral presentations for high (A) and low (B) intensity conditions. Likewise, mean RTs were slower for low- than high-intensity stimulus presentations.

Figure 4. Response Time CDFs and Race-Model Inequalities

(A – D) Step-by-step analysis of response time CDFs for high and low intensities at 0° location. (A and B) Group average multisensory (purple lines) and race-model (black line) CDF for high (A) and low (B) intensity stimuli at the central location. (C) Group average race-model inequality for high and low intensities at 0° location, computed by subtracting race-model CDFs from multisensory CDFs for each subject. (D)Group average race-model violations for high and low intensities at 0° location. Race model violations were calculated by averaging only positive portions of each subject’s race-model inequality. (E – H) CDF analyses shown in (C) and (D) for all stimulus conditions and average areas under these curves. (E) Race-model inequalities for all spatial locations and intensity levels averaged across subjects. (F) Average area under the curve for race-model inequalities for all conditions across subjects. (G) Race-model violation curves for all spatial locations and intensity levels averaged across subjects. Curves decrease in amplitude across space for high intensity stimuli while low intensity curves increase in amplitude across space. (H) Average area under the curve for race model violations, as in (F), showed a significant interaction driven by differences primarily at 60°. Central locations showed no difference for intensity.

Figure 5. Response Time CDF Interactions

(A) CDF contrasts for main effect of intensity were calculated by subtracting each subject’s low intensity race model violation curve from their high intensity curve and averaging across subjects. Positive values represent greater enhancement in high intensity conditions. There is a general trend for the CDF contrasts to be more negative in the periphery. (B,C,D) CDFs from (A) were subtracted to produce interaction CDFs. CDFs were divided into 100ms bins for statistical analysis. Shaded regions represent the 95% confidence intervals. (B) The 0° − 30° interaction yielded no significant bins. (C,D) Significant bins were present in the 0° − 60° and 0° − 90° interactions.