The Role of Frontal Cortical and Medial-Temporal Lobe Brain Areas in Learning a Bayesian Prior Belief on Reversals

Abstract

Reversal learning has been extensively studied across species as a task that indexes the ability to flexibly make and reverse deterministic stimulus-reward associations. Although various brain lesions have been found to affect performance on this task, the behavioral processes affected by these lesions have not yet been determined. This task includes at least two kinds of learning. First, subjects have to learn and reverse stimulus-reward associations in each block of trials. Second, subjects become more proficient at reversing choice preferences as they experience more reversals. We have developed a Bayesian approach to separately characterize these two learning processes. Reversal of choice behavior within each block is driven by a combination of evidence that a reversal has occurred, and a prior belief in reversals that evolves with experience across blocks. We applied the approach to behavior obtained from 89 macaques, comprising 12 lesion groups and a control group. We found that animals from all of the groups reversed more quickly as they experienced more reversals, and correspondingly they updated their prior beliefs about reversals at the same rate. However, the initial values of the priors that the various groups of animals brought to the task differed significantly, and it was these initial priors that led to the differences in behavior. Thus, by taking a Bayesian approach we find that variability in reversal-learning performance attributable to different neural systems is primarily driven by different prior beliefs about reversals that each group brings to the task.

Significance statement: The ability to use prior knowledge to adapt choice behavior is critical for flexible decision making. Reversal learning is often studied as a form of flexible decision making. However, prior studies have not identified which brain regions are important for the formation and use of prior beliefs to guide choice behavior. Here we develop a Bayesian approach that formally characterizes learning set as a concept, and we show that, in macaque monkeys, the amygdala and medial prefrontal cortex have a role in establishing an initial belief about the stability of the reward environment.

Keywords: Bayesian prior; amygdala; learning set; medial prefrontal cortex; orbitofrontal cortex; reversal learning.

Figure 1.

Intended lesion location for the 12 operated groups. Schematic illustration of the brain showing intended lesion locations.

Figure 2.

Object reversal-learning task. For each trial, the monkey displaced one of two objects to gain a food reward underneath. Only one object led to a reward for each trial in a deterministic manner. When performance reached the criterion, the object–reward contingencies switched. The monkeys performed 30 trials each day, until either seven or nine serial reversals occurred.

Figure 3.

Choice behavior of the control group determined by the behavioral choice model (M = 1). A, The posterior probability distribution of all nine reversals. B, The estimated reversal point determined by the behavioral choice model and total errors to criterion for each reversal. The estimated reversal point is subtracted by 61 to allow the zero point to indicate the point of actual reversal.

Figure 4.

Estimated reversal points of all lesion groups and unoperated controls, classified into three clusters. A, Excitotoxic lesion of medial OFC (Walker's area 14) clustered together with the control group, showing no signs of deficit. B, The OFC_ASP, rhinal cortex, hippocampus, and combined unilateral AMG-OFC lesion groups clustered together, showing varying degrees of impairment in detecting reversals. C, The AMG and cingulate region groups (areas 24, 25, 32), as well as groups with excitotoxic lesions of lateral OFC (Walker's area 11/13) and agranular insular cortex showed a tendency of reversing earlier compared with controls in the first few reversals.

Figure 5.

Estimated reversal points and posterior probability distributions averaged across lesion groups for each cluster. A, Cluster 2 showed a strong overall impairment, while Cluster 3 showed an enhancement in learning to reverse, especially in the first few reversals. Regardless, all clusters still learned to reverse earlier with more experience. B, The posterior distribution shows a marked enhancement of cluster 3 in both accuracy and precision during the first reversal.

Figure 6.

Choice behavior aligned at the estimated reversal point shows that monkeys exhibit minimal sampling behavior. Although aligning the choice behavior at the actual reversal point (red) may suggest a reinforcement learning process after the switch, aligning at the estimated reversal point (blue) shows that monkeys rarely sample the other option once they suspect a reversal has occurred. Trial numbers are normalized by creating 10 bins before and after the aligned reversal point.

Figure 7.

Win–stay/lose–shift behavior of the three clusters. A, Cluster 2 displayed less win–stay behavior compared with the other two clusters, but the probability of win–stay remained high and stable across all reversals for all three clusters. B, Lose–shift behavior differed between clusters, but this behavior also increased as the monkeys experienced more reversals.

Figure 8.

Evidence and prior at each reversal point determined by the causal model (M = 2). A, Evidence accumulated regarding whether the reversal had occurred when the monkey decided to reverse its behavior. All clusters learned to reverse with less evidence with more experience. B, Prior belief of the monkey that reversals occur in the environment. As they experience more reversals, all clusters increased their prior belief that reversals occur. Solid lines are data; dashed lines are model fits to data.