Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex

Abstract

Uncertainty about the function of orbitofrontal cortex (OFC) in guiding decision-making may be a result of its medial (mOFC) and lateral (lOFC) divisions having distinct functions. Here we test the hypothesis that the mOFC is more concerned with reward-guided decision making, in contrast with the lOFC's role in reward-guided learning. Macaques performed three-armed bandit tasks and the effects of selective mOFC lesions were contrasted against lOFC lesions. First, we present analyses that make it possible to measure reward-credit assignment--a crucial component of reward-value learning--independently of the decisions animals make. The mOFC lesions do not lead to impairments in reward-credit assignment that are seen after lOFC lesions. Second, we examined how the reward values of choice options were compared. We present three analyses, one of which examines reward-guided decision making independently of reward-value learning. Lesions of the mOFC, but not the lOFC, disrupted reward-guided decision making. Impairments after mOFC lesions were a function of the multiple option contexts in which decisions were made. Contrary to axiomatic assumptions of decision theory, the mOFC-lesioned animals' value comparisons were no longer independent of irrelevant alternatives.

Fig. 1.

Task description and histology. (A) On each trial, three clipart stimuli were presented on a touch-screen; each was associated with a different outcome probability delivered according to the reward schedule. Gray, blue, and red circles represent different 250-ms tones. (B) Example of a varying reward schedule. (C) Medial OFC (Left) and lOFC (Center) lesion locations represented on an unoperated control, with redness indicating lesion overlap (mOFC: one to four animals; lOFC: one to three animals). The full overlap of each lesion type in all animals in each group is shown for mOFC and lOFC groups in blue and pink, respectively, on the same sections (Right).

Fig. 2.

Reward-credit assignment during value learning. (A) Influence of rewards on the current trial on valuation of stimuli chosen in recent trials. The difference in likelihood of choosing option A on trial n after previously selecting option B on trial n-1 as a function of whether or not a reward was received for this choice. Data are plotted based on the length of choice history on A [(Left) one previous choice of A; (Center) two to three previous choices of A; (Right) four to seven previous choices of A]. After lOFC lesions (Inset: redrawn from ref. 9), but not mOFC lesions (main panel), the credit for the outcome (reward or no reward) received for choosing B on the current trial (n) is partly assigned to stimuli chosen in earlier trials. (B) Influence of past rewards on valuation of current choice. The difference in likelihood of choosing option B on trial n after previously selecting option A on trials n-2 to n-5 and option B on the previous trial (n-1), as a function of whether the A choices were or were not rewarded (“?” on axis labels indicates presence or absence of reward at that point in the history). After lOFC lesions (Inset: redrawn from ref. 9), but not mOFC lesions (main panel), credit for rewards received for earlier choices of A is partly assigned to the subsequent choice of B. Preoperative control, post-mOFC lesion, and post-lOFC lesion data are shown in green, blue, and pink, respectively.

Fig. 3.

Effect of option value proximity. (A–C) Fixed reward schedules V2_HIGH (A), V2_MID (B), and V2_LOW (C). Proportion of choices of V1 in the fixed reward schedules, V2_HIGH (D and G), V2_MID (E and H), and V2_LOW (F and I). (D–F) Control pre-lesion (green) and post-mOFC (blue) lesion performance. (G–I) Postoperative lOFC (pink) and matched unoperated control (green) performance (D–F). Insets show number of trials to reach 70% V1 choices. Medial OFC lesions caused impairments in V2_HIGH (A and D) but lOFC lesions impaired performance in V2_MID (B and H) and V2_LOW (C and I). Proportion of V1 choices in the varying reward schedules as a function of V1V2 reward differences. Pre- and postoperative data from the mOFC (J, L, and N) and lOFC group (K, M, and O) from varying schedule trials with V1V2 value differences corresponding to the fixed-schedule value differences in D–F and G–I. (J–O) The proportion of trials on which animals failed to pick V1 so they can be compared readily with the Insets in D–I. Once again, the mOFC impairment was apparent on trials with proximate value options (compare J and H).

Fig. 4.

(A) Weights of influence of stimulus value differences on choices of V1 before (green) and after (blue) mOFC lesion. The interactions between the stimulus value differences, for example the interaction between the V1V2 and V2V3 difference [(V1V2)(V2V3)], and (V1V2)V3 and (V2V3)V3, are shown on the right of the figure. Although the interaction terms are positive before surgery, indicating such combinations of value difference were conducive to good performance, they are significantly more negative after mOFC lesion. (B) Proportion of trials on which monkeys chose options as a function of the value difference with respect to one other option in the context of a high (solid line) or low (dashed line) value third option before (i) and after (ii) mOFC lesion. The same effect remained statistically significant in the mOFC lesion group, even if small in absolute terms, when the analysis focused just on the first halves of each testing session when performance tended to be strong even postoperatively (Ci and Cii), but no such effect was seen after lOFC lesions (Ciii and Civ). In summary, only after mOFC lesions are monkeys’ value comparisons dependent on irrelevant alternatives.