Constructing and Forgetting Temporal Context in the Human Cerebral Cortex

Abstract

How does information from seconds earlier affect neocortical responses to new input? We found that when two groups of participants heard the same sentence in a narrative, preceded by different contexts, the neural responses of each group were initially different but gradually fell into alignment. We observed a hierarchical gradient: sensory cortices aligned most quickly, followed by mid-level regions, while some higher-order cortical regions took more than 10 seconds to align. What computations explain this hierarchical temporal organization? Linear integration models predict that regions that are slower to integrate new information should also be slower to forget old information. However, we found that higher-order regions could rapidly forget prior context. The data from the cortical hierarchy were instead captured by a model in which each region maintains a temporal context representation that is nonlinearly integrated with input at each moment, and this integration is gated by local prediction error.

Keywords: computational modeling; event boundary; fMRI; hierarchy; inter-subject correlation; prediction error; sequence processing; temporal context; temporal integration; timescales.

Figure 1.. Computational models of distributed and hierarchical process memory.

(A) Schematic of experiment and results from Lerner et al. (2011). fMRI participants listened to an intact auditory narrative as well as versions scrambled at the scales of words, sentences and paragraphs. (B) Lower-level regions (e.g. auditory cortex) exhibited responses that were reliable across all stimuli, with little dependence on prior temporal context. By contrast, higher-level regions (e.g. precuneus) exhibited responses that depended at each moment on tens of seconds of prior context in the stimuli. (C) Schematic of the “process memory hierarchy”. Lower-level regions (e.g. sensory regions) exhibit shorter integration timescales, integrating over entities such as phonemes and words. Higher-level regions (e.g. lateral and medial parietal regions) exhibited longer integration timescales, combining information on the scale of entire events (e.g. paragraphs of text). (D) Schematic of predicted data when comparing the representations of brain regions sensitive to temporal context on different scales. The dependent variable is the “intact-scramble correlation”, quantifying the similarity of neural response to the same input in different contexts. (E) Schematic of a signal gain model for explaining the pattern of brain responses shown in panel D. (F) Schematic of hierarchical linear integrator model, HLI. LSS = long scale scramble, MSS = medium scale scramble, FSS = fine scale scramble. HLI = hierarchical linear integrator model.

Figure 2.. Gradual alignment of responses to a common stimulus preceded by different context.

(A) For each sentence, inter-subject pattern correlation (ISPC) was measured by correlating the spatial pattern of activation at each time point across the two groups. (B) ISPC was calculated between one subject and the average of the rest of the subjects within the intact group (rII); or between one subject and the rest of the scrambled group (rSS); or across the intact and scrambled groups (rSI). (C) ISPC analysis for the same sentence preceded by different contexts (DE:CE). Here, sentence E followed sentence D for the Intact group, but it followed sentence C for the Scrambled group. (D) Average ISPC for all sentences in ROIs within an auditory (A1+) region and a right TPJ region. Shaded area indicates a 95% confidence interval on individual rSI estimates. (E) The rII, rSS, and rSI_DE:CE curves are shown for individual regions, grouped by “alignment time”. The individual region curves are pale gray, while mean curves for each group of regions is in thick blue (rII), orange (rSS), and gray (rSI_DE:CE). Note that the rII and rSS curves do not ramp, neither for the mean curve, nor for individual regions, while the rSI curves show ramping in almost all regions. (F) Simulation of rII, rSS and rSI for the signal gain model. The rSI curves exhibit ramping, but the alignment times are stable across levels. (G) Simulation of, rSS and rSI for the HLI model. The alignment time is greater in higher levels of the HLI model. A1 = primary auditory cortex, rTPJ = right temporal-parietal junction.

Figure 3.. Hierarchical timescales of context construction across the human cerebral cortex.

(top) Cortical map of the timescale at which neural responses align to a common input preceded by different contexts. Alignment time is quantified as the time for each rSI_DE:CE curve to reach half its maximum value. (bottom) Fitted logistic curves for four representative ROIs along the cortical hierarchy. A1 = primary auditory cortex, IPL = inferior parietal lobe, STG = superior temporal gyrus, rSI = intact-scramble inter-subject pattern correlation.

Figure 4.. Distinct timescales of alignment and separation in cortical dynamics.

(A) Schematic of internal representations falling into and out of alignment as common and distinct inputs are presented. Two groups gradually construct a shared context when they listen to the same input preceded by different contexts, and thus their neural responses fall into alignment. When common input ends, the two groups begin to process a distinct input preceded by a shared context, and participants forget this shared context over time. (B) Schematic of inter-subject pattern correlation (ISPC) analysis, when different speech segments are preceded by the same context. Here, segment D in the intact group and segment E in the scramble group were both preceded by segment C (CD:CE). (C) Empirical rSI_DE:CE results grouped by alignment time of 3–6 seconds, 6–9 seconds and 9–11 seconds. (D) Empirical rSI_CD:CE results, using the same region groupings from the rSI_DE:CE results in Panel C. Regions at different levels of cortical hierarchy can forget context at similar rates. rSI = intact-scramble ISPC.

Figure 5.. Modeling context construction and context forgetting.

(A) HLI model schematic: the new state of each unit is a linear weighted sum of its old state and its new input. (B) HAT model schematic: each region maintains a representation of temporal context, which is combined with new input to form a simplified joint representation. (C) An AT unit, in which local context CNTX is updated via hidden representation HID and current input IN, modulated by time constant τ and “surprise” α. α is computed via auto-associative error Δ and a scaling parameter k. (D) In HAT, the input to level i is gated by surprise α from level (i-1). (E) HLI simulation of rSI_DE:CE predicts longer alignment time at higher stages of processing. (F) HAT simulation of rSI_DE:CE predicts longer alignment time at higher stages of processing. (G) Empirical rSI_DE:CE results grouped by alignment time, consistent with predictions of both HLI and HAT. (H) HLI simulation of rSI_CD:CE predicts that regions that construct context slowly will also forget context slowly. (I) HAT simulations predict that the timescale of context separation (rSI_CD:CE) need not be slower in levels of the model with longer alignment times (rSI_DE:CE). (J) Empirical rSI_CD:CE results grouped by alignment time. HLI = hierarchical linear integrator, HAT = hierarchical autoencoders in time, AT = autoencoder in time, rSI = intact-scramble ISPC.