A shared representation of order between encoding and recognition in visual short-term memory

Abstract

Many complex tasks require people to bind individual events into a sequence that can be held in short term memory (STM). For this purpose information about the order of the individual events in the sequence needs to be maintained in an active and accessible form in STM over a period of few seconds. Here we investigated how the temporal order information is shared between the presentation and response phases of an STM task. We trained a classification algorithm on the fMRI activity patterns from the presentation phase of the STM task to predict the order of the items during the subsequent recognition phase. While voxels in a number of brain regions represented positional information during either presentation and recognition phases, only voxels in the lateral prefrontal cortex (PFC) and the anterior temporal lobe (ATL) represented position consistently across task phases. A shared positional code in the ATL might reflect verbal recoding of visual sequences to facilitate the maintenance of order information over several seconds.

Keywords: Neuroimaging; Positional code; Sequence position; Short term memory; Temporal position; fMRI.

Fig. 1

Sequence representation and positional code. (A) Representation of two sequences as mappings between item codes and temporal position codes. (B) Left: representation of the temporal position of three first items in a 7-item sequence. The variance around positional code is represented in terms of the darkness of the circle. Right: the item at position two is retrieved by reinstating each positional code which then cues the associated item. (C) Examples of temporal position selective neurons from Berdyyeva and Olson (2010). From left to right: Pre-supplementary motor area neuron selective for 1st position, Supplementary eye field neuron selective for 2nd position, and Supplementary motor area neuron selective for the 3rd position in the serial object task.

Fig. 2

Examples of a single trial: (A) Three-item sequence where all items were presented in the recognition phase, item-order mappings remained the same; (B) two-item sequence without recognition, (C) single item’sequence’ with recognition, item-order mapping not the same. (D) Examples of stimuli.

Fig. 3

Regions where the position of the items within a sequence was decoded significantly above chance across participants: red-yellow – significantly above chance between task phases, blue-cyan – significantly above chance within a presentation task phase only. Note that within-phase and across-phase classification maps are not directly comparable and are overlaid here only for visualisation purposes (see Supplementary Information ”Item position classification accuracy within single task phases” for more information). Abbreviations correspond to the following cortices: MF – middle frontal lobe, pOr – pars orbitalis, pTr – pars triangularis, pOp – pars opercularis, pC – precentral area, ST – superior temporal lobe, MT – middle temporal lobe, IT – inferior temporal lobe, LOC – lateral occipital lobe, SM – supramarginal area.

Fig. 4

Classification accuracy and pattern similarity between two task phases in rostro-lateral prefrontal (A) and anterior temporal regions (B). Bar charts display the average classification accuracy across participants by comparing the known position labels (bar groups) to the predictions made by the classification algorithm (bars within the group). Bars show the proportion of predicted values for each position. Correct classifications are represented with a darker bar. Error bars show the standard error of the mean. The red line depicts the chance level classification accuracy 1/3. Similarity matrices display average pairwise pattern correlations (Pearson's ρ) between two task phases: P – presentation, R – recognition, 1, 2, 3 – position. Cells on the diagonal show the pattern correlation within the same positions between two task phases. Abbreviations: MF – middle frontal lobe, pOr – pars orbitalis, aST – antero-superior temporal lobe, aMT – anterior middle temporal lobe.

Fig. 5

(A) Change in position representations across task phases averaged over participants in prefrontal (blue) and temporal (green) regions of interest. Error bars show the standard error of the mean. Analysis of variance was significant for region (F=10.38, df=3, p<0.001) but not position (F=0.23, df=2, p=0.79). (B) Change in position representations for a single subject between two task phases. For visualisation purposes only values from two most discriminant voxels are plotted. Empty markers represent the distribution of means for the presentation phase, filled markers for the recognition phase. Black – first position, red – second position, blue – third position. Ellipses represent two standard deviations around the mean: dotted ellipses – presentation, solid ellipses – recognition. Straight lines represent the Euclidean distance the mean of the distribution has moved in two-voxel space between presentation and recognition.