The representation of order information in auditory-verbal short-term memory

Abstract

Here we investigate how order information is represented in auditory-verbal short-term memory (STM). We used fMRI and a serial recall task to dissociate neural activity patterns representing the phonological properties of the items stored in STM from the patterns representing their order. For this purpose, we analyzed fMRI activity patterns elicited by different item sets and different orderings of those items. These fMRI activity patterns were compared with the predictions made by positional and chaining models of serial order. The positional models encode associations between items and their positions in a sequence, whereas the chaining models encode associations between successive items and retain no position information. We show that a set of brain areas in the postero-dorsal stream of auditory processing store associations between items and order as predicted by a positional model. The chaining model of order representation generates a different pattern similarity prediction, which was shown to be inconsistent with the fMRI data. Our results thus favor a neural model of order representation that stores item codes, position codes, and the mapping between them. This study provides the first fMRI evidence for a specific model of order representation in the human brain.

Keywords: short-term memory; temporal order; working memory.

Figure 1.

Positional and chaining models of order representation. A, An example how the sequences CAT PIG DOG and DOG CAT PIG are differentially encoded by the positional and chaining models. Numbers refer to the positions of the items (CAT, PIG, DOG) in the sequence. Items are color-coded to highlight the permutations. B, Difference in order encoding yields different pairwise similarity matrices for all possible permutations of the items CAT, PIG, and DOG. We assume that the pairwise similarity between two sequences according to a positional model is given by the Hamming distance between the sequences (see Materials and Methods for details). For the chaining model, pairwise similarity between two sequences can be described by the interitem distance between the sequences. The permutations of the items are number-coded on the axes; e.g., 123 refers to CAT PIG DOG, etc. C, Color-coded similarity matrices.

Figure 2.

Experimental task. A, Memory for the items in a sequence versus memory for the order of the items in a sequence. Items are nonoverlapping sets bisyllabic nonwords. B, Single trial. C, Structure of trials.

Figure 3.

Multivariate analysis of order information. First, we generate pairwise similarity matrices of orderings based on the positional and chaining models using the Hamming and interitem distances respectively. Next, we extract each participant's neural response to each sequence to estimate pairwise response pattern similarities. Pattern similarities are then compared with a predicted similarity structure given by the Hamming and the interitem distances between sequences by computing a correlation coefficient between the two matrices. The probability of the correlation is obtained by using a permutation test where the columns of the matrices was shuffled 10,000 times.

Figure 4.

Item information. Regions where the set of presented items was decoded significantly above chance across participants. Whole-brain FDR threshold of p < 0.05. A, Encoding phase of the task: red–yellow; recall phase of the task: navy–magenta. B, Brain areas where the information during the encoding phase predicted the activity patterns during the recall phase.

Figure 5.

Order information. Regions where the similarity of activity patterns were predicted by the Hamming distance significantly above chance across participants. A, Encoding phase of the task: red–yellow; recall phase of the task: navy–magenta. B, Correlation of order information between different brain areas during the encoding and recall phases: (1) L-vMC, (2) R-IFG, (3) L-SPT, (4) L-IFG, (5) L-IPL, (6) L-SMG, (7) L-IFG, (8) R-SMG, (9) R-IPL, and (10) R-vMC. Only correlations for which p < 0.001 are shown. C, The similarity structure of fMRI activity patterns elicited by different orderings in three ROIs that encoded positional item-order associations. The similarity structure was visualized by performing nonmetric multidimensional scaling (in two dimensions) on the dissimilarity matrix of the multivoxel patterns evoked by all orderings using MATLAB's mdscale function. D, Similarity structure for three ROIs which encoded only item information and where not significantly correlated with the positional model prediction.