Discovering Event Structure in Continuous Narrative Perception and Memory

Abstract

During realistic, continuous perception, humans automatically segment experiences into discrete events. Using a novel model of cortical event dynamics, we investigate how cortical structures generate event representations during narrative perception and how these events are stored to and retrieved from memory. Our data-driven approach allows us to detect event boundaries as shifts between stable patterns of brain activity without relying on stimulus annotations and reveals a nested hierarchy from short events in sensory regions to long events in high-order areas (including angular gyrus and posterior medial cortex), which represent abstract, multimodal situation models. High-order event boundaries are coupled to increases in hippocampal activity, which predict pattern reinstatement during later free recall. These areas also show evidence of anticipatory reinstatement as subjects listen to a familiar narrative. Based on these results, we propose that brain activity is naturally structured into nested events, which form the basis of long-term memory representations.

Keywords: Hidden Markov Model; event model; event segmentation; fMRI; hippocampus; memory; narrative; perception; recall; reinstatement; situation model.

Figure 1. Theory of event segmentation and memory

During perception, events are constructed at a hierarchy of timescales (1), with short events in primary sensory regions and long events in regions including the angular gyrus and posterior medial cortex. These high-level regions have event boundaries that correspond most closely to putative boundaries identified by human observers (2), and represent abstract narrative content that can be drawn from multiple input modalities (3). At the end of a high-level event, the situation model is stored into long-term memory (4) (resulting in post-boundary encoding activity in the hippocampus), and can be reinstated during recall back into these cortical regions (5). Prior event memories can also influence ongoing processing (6), facilitating prediction of upcoming events in related narratives. We test each of these hypotheses using a data-driven event segmentation model, which can automatically identify transitions in brain activity patterns and detect correspondences in activity patterns across datasets.

Figure 2. Event segmentation model

(a) Given a set of (unlabeled) timecourses from a region of interest, the goal of the event segmentation model is to temporally divide the data into “events” with stable activity patterns, punctuated by “event boundaries” at which activity patterns rapidly transition to a new stable pattern. The number and locations of these event boundaries can then be compared across brain regions or to stimulus annotations. (b) The model can identify event correspondences between datasets (e.g. responses to movie and audio versions of the same narrative) that share the same sequence of event activity patterns, even if the duration of the events is different. (c) The model-identified boundaries can also be used to study processing evoked by event transitions, such as changes in hippocampal activity coupled to event transitions in the cortex. (d) The event segmentation model is implemented as a modified Hidden Markov Model (HMM) in which the latent state s_t for each timepoint denotes the event to which that timepoint belongs, starting in event 1 and ending in event K. All datapoints during event k are assumed to be exhibit high similarity with an event-specific pattern m_k. See also Figs. S1, S2, S3.

Figure 3. Event segmentation model for movie-watching data reveals event timescales

The event segmentation model identifies temporally-clustered structure in movie-watching data throughout all regions of cortex with high intersubject correlation. The optimal number of events varied by an order of magnitude across different regions, with a large number of short events in sensory cortex and a small number of long events in high-level cortex. For example, the timepoint correlation matrix for a region in the precuneus exhibited coarse blocks of correlated patterns, leading to model fits with a small number of events (white squares), while a region in visual cortex was best modeled with a larger number of short events (note that only ~3 minutes of the 50 minute stimulus are shown, and that the highlighted searchlights were selected post-hoc for illustration). The searchlight was masked to include only regions with intersubject correlation > 0.25, and voxelwise thresholded for greater within- than across-event similarity, q<0.001. See also Figs. S3, S4, S8.

Figure 4. Cortical event boundaries are hierarchically structured and are related to human-labeled event boundaries, especially in posterior medial cortex

(a) An example of boundaries evoked by the movie over a four-minute period shows how the number of boundaries decreases as we proceed up the hierarchy, with boundaries in posterior medial cortex most closely related to human annotations of event transitions. (b) Event boundaries in higher regions are present in lower regions at above-chance levels (especially pairs of regions that are close in the hierarchy), suggesting that event segmentation is in part hierarchical, with lower regions subdividing events in higher regions. (c) All four levels of the hierarchy show an above-chance match to human annotations (the null distribution is shown in gray), but the match increases significantly from lower to higher levels (^† p=0.058, * p<0.05, ** p<0.01).

Figure 5. Movie-watching model generalizes to audio narration in high-level cortex

After identifying a series of event patterns in a group of subjects who watched a movie, we tested whether this same series of events occurred in a separate group of subjects who heard an audio narration of the same story. The movie and audio stimuli were not synchronized and differed in their duration. We restricted our searchlight to voxels that responded to both the movie and audio stimuli (having high intersubject correlation within each group). Movie-watching event patterns in early auditory cortex (dotted line) did not generalize to the activity evoked by audio narration, while regions including the angular gyrus, temporoparietal junction, posterior medial cortex, and inferior frontal cortex exhibited shared event structure across the two stimulus modalities. This analysis, conducted using our data driven model, replicates and extends the previous analysis of this dataset (Zadbood et al., 2016) in which the event correspondence between the movie and audio narration was specified by hand. The searchlight is masked to include only regions with intersubject correlation > 0.1 in all conditions, and voxelwise thresholded for above-chance movie-audio fit, q<10⁻⁵. See also Fig. S8.

Figure 6. Hippocampal activity increases at cortically-defined event boundaries

To determine whether event boundaries may be related to long-term memory encoding, we identify event boundaries based on a cortical region and then measure hippocampal activity around those boundaries. In a set of regions including angular gyrus, posterior medial cortex, and parahippocampal cortex, we find that event boundaries robustly predict increases in hippocampal activity, which tends to peak just after the event boundary. The searchlight is masked to include only regions with intersubject correlation > 0.25, and voxelwise thresholded for post-boundary hippocampal activity greater than pre-boundary activity, q<0.001. See also Figs. S5, S8.

Figure 7. Movie-watching events are reactivated during individual free recall, and reactivation is related to hippocampal activation at encoding event boundaries

(a) We can obtain an estimated correspondence between movie-watching data and free-recall data in individual subjects by identifying a shared sequence of event patterns, shown here for an example subject using data from posterior cingulate cortex. (b) For each region of interest, we tested whether the movie and recall data shared an ordered sequence of latent events (relative to a null model in which the order of events was shuffled between movie and recall). We found that both angular gyrus (blue) and posterior cingulate cortex (green) showed significant reactivation of event patterns, while early auditory cortex (red) did not. (c–d) Events whose offset drove a strong hippocampal response during encoding (movie-watching) were strongly reactivated for longer fractions of the recall period, both in the angular gyrus and the posterior cingulate. Error bars for event points denote s.e.m. across subjects, and error bars on the best-fit line indicate 95% confidence intervals from bootstrapped best-fit lines. See also Fig. S6.

Figure 8. Prior memory shifts movie-audio correspondence

The event segmentation model was fit simultaneously to a data from a group watching the movie, the same group listening to the audio narration after having seen the movie (“memory”), and a separate group listening to the audio narration for the first time (“no memory”). By examining which timepoints were estimated to fall within the same latent event, we obtained a correspondence between timepoints in the audio data (for both groups) and timepoints in the movie data. We found regions in which the correspondence in both groups was close to the human-labeled correspondence between the movie and audio stimuli (black boxes), but the memory correspondence (orange) significantly led the non-memory correspondence (blue) (indicated by an upward shift on the correspondence plots; note that the highlighted searchlights were selected post-hoc for illustration). This suggests that cortical regions of the memory group were anticipating events in the narration based on knowledge of the movie. The searchlight is masked to include only regions with intersubject correlation > 0.1 in all conditions, and voxelwise thresholded for above-chance differences between memory and no memory groups, q<0.05. See also Fig. S7, S8.