Memory and learning with rapid audiovisual sequences

Abstract

We examined short-term memory for sequences of visual stimuli embedded in varying multisensory contexts. In two experiments, subjects judged the structure of the visual sequences while disregarding concurrent, but task-irrelevant auditory sequences. Stimuli were eight-item sequences in which varying luminances and frequencies were presented concurrently and rapidly (at 8 Hz). Subjects judged whether the final four items in a visual sequence identically replicated the first four items. Luminances and frequencies in each sequence were either perceptually correlated (Congruent) or were unrelated to one another (Incongruent). Experiment 1 showed that, despite encouragement to ignore the auditory stream, subjects' categorization of visual sequences was strongly influenced by the accompanying auditory sequences. Moreover, this influence tracked the similarity between a stimulus's separate audio and visual sequences, demonstrating that task-irrelevant auditory sequences underwent a considerable degree of processing. Using a variant of Hebb's repetition design, Experiment 2 compared musically trained subjects and subjects who had little or no musical training on the same task as used in Experiment 1. Test sequences included some that intermittently and randomly recurred, which produced better performance than sequences that were generated anew for each trial. The auditory component of a recurring audiovisual sequence influenced musically trained subjects more than it did other subjects. This result demonstrates that stimulus-selective, task-irrelevant learning of sequences can occur even when such learning is an incidental by-product of the task being performed.

Figure 1

Schematic examples of Nrep stimuli presented on different trials.

Figure 2

(A) Values of d′ produced with Congruent and Incongruent stimuli. The symbol * signifies that performance with Congruent stimuli significantly exceeds performance with Incongruent stimuli. (B) Mean proportion of Repeat responses for each stimulus type. Note that for Rcon and Rincon stimuli, a Repeat response was correct, while for the other three stimulus types a Repeat response was incorrect. The leftmost * symbol indicates that performance is significantly higher with Rcon stimuli than with Rincon stimuli; the rightmost * symbol signifies that performance with Nrep stimuli is significantly worse than all other stimulus types with the exception of Rincon. Error bars represent between-subjects standard errors.

Figure 3

Proportion of correct responses in Experiment 1 with Rincon stimuli of varying degrees of auditory “repeatedness” in 10 equally populous bins. The degree of repetition in the auditory component of a Rincon stimulus is calculated by taking the mean difference between items n and n + 4 for n = 1:4. The resulting value was normalized so that a repeatedness value of 1 would correspond to a perfect repeat. Vertical bars represent 95% confidence intervals, while horizontal bars show the range of values in a bin. Also plotted (▪) is the mean proportion of correct responses with Rcon stimuli. The overlap in confidence intervals between the Rcon stimuli and the bin of Rincon stimuli closest to perfectly repeated provides evidence for the claim that the effect of congruency can be accounted for by the additional Repeat information in the auditory component.

Figure 4

Distribution of absolute differences between the luminances of successive items for Nincon trials in Experiment 1, divided into 10 equally spaced bins. The vertical line represents the background luminance, 19.03 cd/m².

Figure 5

Examples of visual components of Non-Repeat, Repeat, and Frozen stimuli. See text for details.

Figure 6

Diagram showing exemplars of Frozen Incongruent sequences. Each subject's Frozen Incongruent stimulus (FRincon) recurred identically and intermittently throughout the experiment. A subject's FRincon stimulus was changed only in Block 3 and only for subjects in the Crossover-Switchback condition. Each FRincon stimulus comprised a Repeat visual sequence accompanied by an incongruent auditory sequence. For subjects assigned to the Crossover-Switchback condition, the sequence's auditory component was switched in Block 3 to a different, incongruent auditory sequence. In contrast, subjects assigned to the Constant condition continued to receive the same FRincon sequence throughout.

Figure 7

Performance (expressed as mean d′ values) with Frozen and Repeat stimuli, in Congruent and Incongruent conditions. As in Experiment 1, performance with Congruent stimuli exceeds performance with Incongruent stimuli. Moreover, performance with Frozen exemplars exceeds performance with Repeat stimuli, and Musicians outperform Non-Musicians on average for all stimulus types. Error bars represent between-subjects standard errors.

Figure 8

Performance with FRincon stimuli by subjects in the Constant and Crossover-Switchback groups. While subjects in the Constant condition show a small, statistically nonsignificant change in d′ over the experiment's four blocks, subjects in the Crossover-Switchback group show a sharp decrease in Block 3. Error bars represent between-subject standard errors.

Figure 9

(A) Musicians' mean performance (expressed as d′) with FRincon stimuli in the Crossover-Switchback and Constant conditions. (B) Non-Musician's mean performance with FRincon stimuli in the Crossover-Switchback and Constant conditions. Note the use of two separate y-axes in each panel, with each axis adjusted such that performance in Block 1 lies at the same point in the plot for both groups of subjects. Error bars in both panels represent between-subjects standard errors.

Figure 10

Proportion of correct responses in Experiment 2 with auditory components of Rincon stimuli of varying “repeatedness” in 10 equally populous bins. The degree of repetition in the auditory component of a particular Rincon stimulus is calculated by taking the mean difference between items n and n + 4 for n = 1:4. The mean repeatedness within each bin were transformed such that a repeatedness value of 1 corresponds to a perfect repeat. Vertical bars denote 95% confidence intervals, while horizontal bars denote the range of each bin. Also plotted (▪) is the mean proportion of correct responses with Rcon stimuli.

Figure 11

Proportion correct versus AUC. Each data point represents one subject. Also shown is the best fitting regression line and the 95% confidence region around that line. Note that because the stimuli were Nincon, a response of “Non-Repeat” was correct.