Learning of distant state predictions by the orbitofrontal cortex in humans

Abstract

Representations of our future environment are essential for planning and decision making. Previous research in humans has demonstrated that the hippocampus is a critical region for forming and retrieving associations, while the medial orbitofrontal cortex (OFC) is an important region for representing information about recent states. However, it is not clear how the brain acquires predictive representations during goal-directed learning. Here, we show using fMRI that while participants learned to find rewards in multiple different Y-maze environments, hippocampal activity was highest during initial exposure and then decayed across the remaining repetitions of each maze, consistent with a role in rapid encoding. Importantly, multivariate patterns in the OFC-VPFC came to represent predictive information about upcoming states approximately 30 s in the future. Our findings provide a mechanism by which the brain can build models of the world that span long-timescales to make predictions.

Fig. 1

Learning-phase task design and behavioral results. a Learning-phase maze trial structure. Across eight unique maze sequences, participants learned which action to make in state 1 in order to receive a deterministic reward at the end of the maze. Participants made a left or right choice in the first state and then proceeded through instructed left or right choices in state 2 and state 3, followed by reward or loss in the feedback state. Maze-unique stimuli from three categories (faces, scenes, and objects) were presented in state 1 and state 3. The delay between state 1 and state 3 was on average greater than 30 s, while repetitions of unique mazes were separated by at least 4 min. Critical decoding analyses focused on representations of information about state 3 at the onset of state 1 (represented by “Decoding” in the blue box). b Mean learning-phase performance across four repetitions of each maze. Half of initial repetitions ended in reward and half in loss, and the resulting mean 50% level of performance is indicated by the open square at repetition 1. Error bars represent standard error of the mean. c Illustration of the participant’s screen view, a cartoon room with potential left and right door options followed by a path through a hallway between states. In state 2 and state 3, the instructed choice was indicated by a shift in the central stimulus to the instructed door. A localizer phase, which was used to derive classifiers for faces, scenes, and objects, followed the learning phase

Fig. 2

Learning-related univariate fMRI responses. a Across repetition of mazes during the learning phase, activity in the hippocampus significantly decreased, supporting a role for the hippocampus in maze encoding (right and left hippocampus, p < 0.05 SVC; image threshold p < 0.005 unc.). b At feedback, beginning >40 s after the start of a maze, activation in the ventral striatum and OFC - ventromedial PFC, among other regions, was significantly activated by the receipt of a reward versus loss. (Images p < 0.05 whole-brain FWE-corrected; full maps available at: https://neurovault.org/images/100616/ and https://neurovault.org/images/100619/)

Fig. 3

Multivariate OFC-VMPFC responses at state 1 onset related to the representation of anticipated future states. a Outline of the OFC-VMPFC region of interest shown over the across-participant average anatomical MRI. b Regression coefficients for decoded activity in the OFC-VMPFC for the current state (state 1), future state (state 3), the repetition since performance was correct, the interaction of current state with correct repetition, and the critical interaction of future state with correct repetition. Distribution density is represented by violin plot width; individual participant datapoints are in black. (*p < 0.05, t test; error bars represent standard error of the mean; statistical comparisons were not completed on the current state representation in (b) because the plotted data represent the results after exclusion of two participants with poor decoding during learning.) c Breakdown of the future state by correct repetition interaction, for illustration. Future state classification is plotted separately for each correct maze repetition; these effects were derived from control models examining current and future state separately for each correct repetition bin