A psychophysical investigation of differences between synchrony and temporal order judgments

Abstract

Background: Synchrony judgments involve deciding whether cues to an event are in synch or out of synch, while temporal order judgments involve deciding which of the cues came first. When the cues come from different sensory modalities these judgments can be used to investigate multisensory integration in the temporal domain. However, evidence indicates that that these two tasks should not be used interchangeably as it is unlikely that they measure the same perceptual mechanism. The current experiment further explores this issue across a variety of different audiovisual stimulus types.

Methodology/principal findings: Participants were presented with 5 audiovisual stimulus types, each at 11 parametrically manipulated levels of cue asynchrony. During separate blocks, participants had to make synchrony judgments or temporal order judgments. For some stimulus types many participants were unable to successfully make temporal order judgments, but they were able to make synchrony judgments. The mean points of subjective simultaneity for synchrony judgments were all video-leading, while those for temporal order judgments were all audio-leading. In the within participants analyses no correlation was found across the two tasks for either the point of subjective simultaneity or the temporal integration window.

Conclusions: Stimulus type influenced how the two tasks differed; nevertheless, consistent differences were found between the two tasks regardless of stimulus type. Therefore, in line with previous work, we conclude that synchrony and temporal order judgments are supported by different perceptual mechanisms and should not be interpreted as being representative of the same perceptual process.

Figure 1. Stimulus illustrations, timing characteristics and subjective impressions of task difficulty.

(A) Top and bottom rows illustrate the visual information and auditory waveform for each stimulus, respectively. (BF) beep-flash stimulus, consisted of a single flash of a white dot on a black background and a single beep. (BFV) beep-flash-constant visual, was the same as (BF) except there was a smaller white dot, illustrated by the dashed circle, constantly on screen. (BFD) beep-flash-drumming, was the same as (BF) except there were 9 beep-flashes in each stimulus. (PLD) point-light-drumming, shown is a single movie frame and the waveform drumbeat. (FV) A single frame from the face-voice movie and the waveform representation of the word ‘tomorrow’. Note, the images are not to scale, the area of the point-light-drummer and the white flash dots (BF, BFV & BFD) approximately subtended the area of the mouth region in FV. (B) Illustration of timing characteristics (onset, offset and duration) for BF and BFV (left), BFD and PLD (middle) and FV (right). COA levels (ms), both audio- and video-leading, are displayed on the y-axis and the x-axis represents duration (ms) – note that x-axis scales are different across the 3 figures. For each COA level gray bars represent overall stimulus durations (see Stimuli section for details), while black bars represent relative timing characteristics between audio and video sequences. Top and bottom black bars can represent either the audio or video sequence. For example, the top black bar of the 67 COA condition for BF would represent the audio sequence for an audio-leading condition, while it would represent the video sequence for a video-leading condition. Vertical black bars in the middle figure (BFD &PLD) highlight that the duration of all COA stimuli was 3s and that the stimuli were cut from a larger audiovisual movie after separating the audio and video cues in time by each COA. (C) Separately for each stimulus type, pie charts represent the percentage of participants who found either SJ or TOJ the most difficult. Order of pie charts matches the order of the stimuli presented above in (A).

Figure 2. Examples of excluded TOJ data for BFD, PLD and FV stimuli.

Red triangles represent the proportion of video first responses at each COA level. Each row displays 3 randomly selected excluded data sets plus the mean of the excluded data for a particular stimulus type (top for BFD, middle for PLD and bottom for FV). The dashed horizontal line represents chance performance. Errorbars are ± one standard error of the mean. BF and BFV conditions are not included as no TOJ/BF data was excluded and only a single subject was excluded from TOJ/BFV.

Figure 3. Mean response proportions and fitted functions.

Each graph presents data from a different stimulus type, defined in the graph title. Mean proportion of synchronous responses (blue squares for SJ) and video first responses (red triangles for TOJ) are plotted for each COA level along with their corresponding best fitting Gaussian functions. The PSS derived from each fit is indicated by appropriately coloured (blue  =  SJ, red  =  TOJ) vertical dashed lines. Errorbars are ± one standard error of the mean.

Figure 4. Correlation across stimulus type and task for PSS

(4A) and TIWs (4B). Significant Spearman correlations (p<0.05) are displayed as coloured boxes (representing rho correlation values), while non-significant correlations were greyed out.

Figure 5. Mean percentage correct for audio-leading and video-leading asynchrony levels.

Top and bottom rows display the SJ/stimulus and TOJ/stimulus combinations, respectively. Each graph presents the mean percentage correct for audio leading (orange) and video leading (green) conditions at each cue onset asynchrony (COA). Errorbars represent 95% confidence intervals calculated from 10,000 bootstraps.