Specializations for reward-guided decision-making in the primate ventral prefrontal cortex

Abstract

The estimated values of choices, and therefore decision-making based on those values, are influenced by both the chance that the chosen items or goods can be obtained (availability) and their current worth (desirability) as well as by the ability to link the estimated values to choices (a process sometimes called credit assignment). In primates, the prefrontal cortex (PFC) has been thought to contribute to each of these processes; however, causal relationships between particular subdivisions of the PFC and specific functions have been difficult to establish. Recent lesion-based research studies have defined the roles of two different parts of the primate PFC - the orbitofrontal cortex (OFC) and the ventral lateral frontal cortex (VLFC) - and their subdivisions in evaluating each of these factors and in mediating credit assignment during reward-based decision-making.

Figure 1:

a| Medial (left) and lateral (right) views of the macaque brain showing the locations of the orbitofrontal cortex (OFC), ventrolateral frontal cortex (VLFC), amygdala, mediodorsal thalamus and indicating selected connections between these regions. Dashed lines indicate structures buried deep in the brain. The OFC extends from the fundus of the lateral orbital sulcus to the rostral sulcus on the medial aspect of the hemisphere. The VLFC extends from just below the principal sulcus, laterally, to the fundus of the lateral orbital sulcus, which forms its shared boundary with the OFC. Caudally, the VLFC is bounded by the fundus of the inferior limb of the arcuate sulcus. Rostrally, both the OFC and the VLFC extend to their boundary with the frontal pole cortex^,. b| Medial (left) and ventral (right) views of the frontal lobe of the macaque brain showing approximate locations of the medial OFC, the lateral OFC and the VLFC. Numerals correspond to cytoarchitectonic areas of Walker. The lateral OFC can be subdivided into anterior and posterior zones, corresponding to cytoarchitectonic areas 11 and 13, respectively. Arrows indicate direct anatomical connections from the agranular orbital and insular cortex to the medial and lateral OFC.

Figure 2:. Selected behavioral tasks used to assess value-based decision making.

a| The 3-armed bandit task. At the start of each session, monkeys are presented with three novel images (options) on a touch screen monitor. Monkeys select one of the images by touching it on the screen. The same three images appear on each trial of the 300-trial session. The delivery of a reward for the selection of each image is predetermined based on four different probabilistic schedules, an example of which is shown in the plot at the right. In this case, over the first 150 trials, option A is the best choice; however as the monkeys approach trial 150 option B becomes the better option and remains so for the next 150 trials. To perform well, monkeys must discriminate the images, sample the outcome associated with each image by choosing it, and track the likelihood of rewards that they received for choosing each image during a test session. Because the reward probabilities assigned to each image changed over the course of the session, the monkeys needed to continually update their valuations of the images. b| A devaluation task. In one training trial, the monkeys learn to discriminate two objects, one of which—the blue cone—covers a food reward. In another training trial, the monkeys learn that a different object—a green hemisphere—is associated with a different food reward. In practice there were 60 pairs of objects and therefore a total of 60 training trials per session, 30 trials with each type of food reward. In the test phase of the task, the monkey consumes one of the two foods to satiety and must then make a choice between two previously rewarded objects. Each test comprised 30 such trials, each with a different pair of objects. This paradigm measures the ability of the monkeys to link objects with the current value of the food, which is influenced by the degree of satiety.

Figure 3:. Effects of selective, excitotoxic lesions of the OFC and VLFC on availability- or desirability-based choices.

a,b| Performance on 3-armed bandit task (FIG. 2) by monkeys with OFC lesions and unoperated controls (a) and monkeys before and after VLFC lesions (b). Colored dots represent the probability of reward associated with choice of the best (high reward) option across a 300-trial session (the best option changes across the session). Colored lines and shaded areas show mean and SEM probability of choice of the high reward option by monkeys with OFC lesions, unoperated controls, monkeys before VLFC lesions and monkeys after VLFC lesions and reflect the ability to track the changing reward probabilities associated with each choice option. The scores for the OFC lesion and control groups overlap, whereas the score is lower for the VLFC lesion group. Bars to the right of the plots show group mean probability of choosing the high reward option in last 150 trials of session. c| Performance on the devaluation task (FIG. 2b) by monkeys with OFC lesions, VLFC lesions and unoperated controls. The graph indicates the proportion of the monkey’s choices that shifted to selection of the object associated with the nonsated food reward, when compared to choices in a baseline condition without satiation, and reflects the monkeys’ ability to make adaptive choices after one food is devalued by selective satiation. The higher the score, the greater the ability of monkeys to choose objects overlying the higher value (nonsated) food. Monkeys with OFC lesions make significantly fewer adaptive choices relative to monkeys in both other groups. d|. Summary of the effects of OFC and VLFC lesions: the difference in the score of lesioned groups from those of comparison groups derived from the contingent learning analysis (3-armed bandit) and the proportion shifted scores from the devaluation task are indicated. Plot highlights the double dissociation of function between two ventral prefrontal cortical areas, i.e., the selective and independent contributions of OFC and VLFC to different types of value updating. Data from Rudebeck and colleagues. Abbreviations: Con, unoperated control monkeys; OFC lesion, monkeys with bilateral excitotoxic lesions of the orbitofrontal cortex; Before VLFC, monkeys before lesions of the ventral lateral frontal cortex; After VLFC, monkeys with bilateral excitotoxic lesions of the ventral lateral frontal cortex.

Figure 4:. Independent contributions of medial and lateral OFC to value-based decision making.

a,b|. Macaques took part in a three-arm bandit task (FIG. 2) in which the value of the best (V1) and worst (V3) choice options are stable within and across the sessions. The value of choice option V2 was stable within sessions but varied across sessions; in the example session shown the value of option V2 was close to (but less than) the value of option V1. The charts show the effects of aspiration lesions of the medial OFC (a) and the lateral OFC (b) on performance. Colored lines and shaded areas show mean and SEM probability of choice of the best value option (V1). Monkeys with lesions of the medial OFC (but not those with lesions of the lateral OFC) were poor at choosing the best option when the value of V2 was close to the value of V1, suggesting that medial OFC contributes to value comparisons. c| Effects of selective, excitotoxic lesions of the lateral or medial OFC on the devaluation task (FIG. 2). The difference score reflects the extent to which the monkeys shift their choices of objects after selective satiation relative to baseline conditions without satiation. The higher the score the greater the number of choices of objects overlying the higher value (nonsated) food. Controls show robust difference scores, indicating their sensitivity to the value of the outcome that is associated with object choices. Lateral OFC lesions result in long-term disruption of this capacity. Medial OFC lesions cause a transient impairment, as illustrated in measurements taken during an early postoperative test; however, the choices of these animals return to control values by the second postoperative test (late). Data from Rudebeck and Murray. d| Effects of reversible GABAergic agonist-induced inactivations of the anterior (area 11) and posterior (area 13) OFC on adaptive choices, assessed using the devaluation task. The proportion choice shifted score reflects the monkeys’ ability to make adaptive choices after one food is devalued by selective satiation. The higher the proportion shifted score, the greater the ability of monkeys to choose objects associated with the higher value (nonsated) food. Inactivation of area 13 during (but not after) selective satiation disrupted adaptive choices, reflecting an impairment in value updating. Inactivation of area 11 after (but not before) selective satiation disrupted adaptive choices, reflecting an impairment in goal selection. Symbols show scores of individual monkeys. Data from Murray et al.

Figure 5:. Effect of amygdala lesions on value coding in the OFC.

Monkeys performed a choice task with a set of familiar stimulus-outcome associations. Different stimuli were associated with different amounts of a juice reward. On each trial, two images (S1 and S2) were presented sequentially and then monkeys were allowed to choose one of the two stimuli. The amount of juice assigned to the chosen image (S1 or S2) was delivered a short time later. a|. Percentage of OFC neurons encoding the magnitude of the associated reward while the monkeys viewed S1 and S2. Data are averaged over the entire population of recorded neurons to illustrate changes in encoding during the period stimuli were being evaluated (0–2000 ms.). Data are aligned to the onset of the presentation of the first image (S1). A bilateral excitotoxic amygdala lesion led to a reduction in the encoding of the value of the anticipated outcome. b|. Percentage of OFC neurons encoding the magnitude of reward near the time of choice and reward delivery. Data are aligned to the onset of reward delivery. A bilateral excitotoxic amygdala lesion led to a reduction in the encoding of the value of the chosen image and of the received reward. Data from Rudebeck and colleagues.