Spatiotemporal integration of looming visual and tactile stimuli near the face

Abstract

Real-world objects approaching or passing by an observer often generate visual, auditory, and tactile signals with different onsets and durations. Prompt detection and avoidance of an impending threat depend on precise binding of looming signals across modalities. Here we constructed a multisensory apparatus to study the spatiotemporal integration of looming visual and tactile stimuli near the face. In a psychophysical experiment, subjects assessed the subjective synchrony between a looming ball and an air puff delivered to the same side of the face with a varying temporal offset. Multisensory stimuli with similar onset times were perceived as completely out of sync and assessed with the lowest subjective synchrony index (SSI). Across subjects, the SSI peaked at an offset between 800 and 1,000 ms, where the multisensory stimuli were perceived as optimally in sync. In an fMRI experiment, tactile, visual, tactile-visual out-of-sync (TVoS), and tactile-visual in-sync (TViS) stimuli were delivered to either side of the face in randomized events. Group-average statistical responses to different stimuli were compared within each surface-based region of interest (sROI) outlined on the cortical surface. Most sROIs showed a preference for contralateral stimuli and higher responses to multisensory than unisensory stimuli. In several bilateral sROIs, particularly the human MT+ complex and V6A, responses to spatially aligned multisensory stimuli (TVoS) were further enhanced when the stimuli were in-sync (TViS), as expressed by TVoS < TViS. This study demonstrates the perceptual and neural mechanisms of multisensory integration near the face, which has potential applications in the development of multisensory entertainment systems and media.

Keywords: 4D film; approaching threats; binding problem; fMRI; multisensory integration; neurocinematics; peripersonal space; psychophysics.

Figure 1

Experimental setup and stimuli. (a) Multisensory stimulation apparatus in psychophysical and fMRI experiments. (b) Spatially aligned looming visual and tactile stimuli near either side of the face. The trace of a looming ball was created by superimposing key frames between 0 and 900 ms, where only one ball was visible per frame (see Supporting Information Figure S1a). (c) Time courses of the eccentricity and diameter of a looming ball. The ball was truncated between 883 and 933 ms (dashed segments) before disappearing at the screen edge [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 2

Experimental paradigms and stimulus timelines. (a) Three representative pairs of multisensory stimuli with different temporal offsets (100, 500, and 1,000 ms) in the psychophysical experiment. (b) Unisensory and multisensory stimuli in the fMRI experiment. Black square: duration of an air puff; Gray bar: duration of a looming ball

Figure 3

Results of the psychophysical experiment. (a) SSI curves of a representative subject. Upper‐left panel: The bottom and top of each box respectively represent the first and third quartiles of the distribution of 20 SSIs. The height of each box indicates the interquartile range (IQR). Each whisker indicates 1.5 IQR above or below the box. The SSI‐median curve connects the mark (median) within each box. Δ: peak. **: significant decrease from the peak, p < .01, Bonferroni corrected. N: insignificant decrease from the peak. (b) SSI curves (thin gray curves) of 20 subjects, overlaid with group mean (a thick black curve) ± standard deviation (thin black curves) in each panel. (c) Correlation coefficients (r) between 20 pairs of SSI‐median and SSI‐IQR curves and between 20 pairs of SSI‐mean and SSI‐s.d. curves in (b)

Figure 4

Results of behavioral data modeling. (a) Joint probability of T (Offset) and S (SSI), $p\left(T,\hspace{0.17em}S\right)$. The color bar represents the joint probability of each $\left(t_m,\hspace{0.17em}{s}_n\right)$ pair. Each row of the 2D plot represents a curve of the joint probability from a bird's eye view. (b) The probability of an SSI, $s_n$, conditioned on a temporal offset, $t_m$: $p\left(S|T=t_m\right)$ (thin curves); within‐group conditional probability: $p\left(S|T\in \mathrm{\tau }\right)$ (thick curves). Each sub‐plot represents a group obtained by applying the k‐means algorithm to $p\left(S|T=t_m\right)$ curves. (c) Curves of cumulative distribution $F\left(S\le s_n|T\in \mathrm{\tau }\right)$ matching the groups τ₁ to τ₃ in (b) [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 5

Group‐average statistical maps in response to right‐face stimuli in the fMRI experiment. Black contours in the left hemisphere (LH): sROIs outlined in response to right‐face stimuli (Figure 5). Cyan contours in the right hemisphere (RH): sROIs outlined in response to left‐face stimuli (Figure 6). Green contours in both hemispheres: brain regions with significant activations across 11 subjects (t ₍₁₀₎ > 2.76, one‐tailed; p < .01, uncorrected). Tactile and visual maps use one color scale, and TVoS and TViS maps use another color scale. Yellow‐red color bar: activation. Cyan‐blue color bar: deactivation. CSv: cingulate sulcus visual area (Smith et al., 2012). RSC: retrosplenial cortex (Huang & Sereno, 2013). Other abbreviations as in text [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 6

Group‐average statistical maps in response to left‐face stimuli in the fMRI experiment. All conventions follow Figure 5 [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 7

Box plots of group‐average F‐statistic distributions in response to unisensory and multisensory event types in ten bilateral sROIs (see Figures 5 and 6). For each sROI, black and gray box plots respectively indicate the responses to contralateral and ipsilateral stimuli. The top and bottom of each box respectively represent F _Q3 and F _Q1. The mark within each box indicates median. The height of each box indicates IQR (F _Q3 − F _Q1). Each whisker indicates 1.5 IQR above or below the box. Each inequality symbol compares the F _Q3 values between two adjacent event types (contralateral stimuli only; see Supporting Information Tables S3 and S4). Each F‐statistic value on the left y‐axis corresponds to a p‐value (Bonferroni‐corrected, n = 11) estimated using a corresponding value (#) on the right y‐axis, for example, p = 10^(−1.76) = 0.0174

Figure 8

Box plots of group‐average F‐statistic distributions in response to unisensory and multisensory event types in eight bilateral and two unilateral sROIs (see Figures 5 and 6). Note that a different scale is used for the occipital cluster (O.C.). Other conventions follow Figure 7