Modulation of value-based decision making behavior by subregions of the rat prefrontal cortex

Abstract

Rationale: During value-based decision-making, organisms make choices on the basis of reward expectations, which have been formed during prior action-outcome learning. Although it is known that neuronal manipulations of different subregions of the rat prefrontal cortex (PFC) have qualitatively different effects on behavioral tasks involving value-based decision-making, it is unclear how these regions contribute to the underlying component processes.

Objectives: Assessing how different regions of the rodent PFC contribute to component processes of value-based decision-making behavior, including reward (or positive feedback) learning, punishment (or negative feedback) learning, response persistence, and exploration versus exploitation.

Methods: We performed behavioral modeling of data of rats in a probabilistic reversal learning task after pharmacological inactivation of five PFC subregions, to assess how inactivation of these different regions affected the structure of responding of animals in the task.

Results: Our results show reductions in reward and punishment learning after PFC subregion inactivation. The prelimbic, infralimbic, lateral orbital, and medial orbital PFC particularly contributed to punishment learning, and the prelimbic and lateral orbital PFC to reward learning. In addition, response persistence depended on the infralimbic and medial orbital PFC. As a result, pharmacological inactivation of the infralimbic and lateral orbitofrontal cortex reduced the number of reversals achieved, whereas inactivation of the prelimbic and medial orbitofrontal cortex decreased the number of rewards obtained. Finally, using simulated data, we explain discrepancies with a previous study and demonstrate complex, interacting relationships between conventional measures of probabilistic reversal learning performance, such as win-stay/lose-switch behavior, and component processes of value-based decision-making.

Conclusions: Together, our data suggest that distinct components of value-based learning and decision-making are generated in medial and orbital PFC regions, displaying functional specialization and overlap, with a prominent role of large parts of the PFC in negative feedback processing.

Keywords: Behavioral modeling; Decision-making; Prefrontal cortex; Punishment; Rats; Reinforcement learning; Reward; Value.

Fig. 1

Effects of PFC inactivation on probabilistic reversal learning. a Probabilistic reversal learning setup. b Example session of one rat. c Effects of PFC inactivation on probabilistic reversal learning. ACC, n = 10 rats; PrL, n = 12 rats; IL, n = 9; mOFC, n = 9; lOFC, n = 9. *P < 0.05, **P < 0.01, ***P < 0.001, ****P < 0.0001 (post hoc Holm-Sidak test; see also the Supplementary statistics table in Online Resource 1; for infusion sites see Online Resource 3)

Fig. 2

Behavioral model selection. a We fit several reinforcement learning models to our data and estimated which model (i.e., strategy) best described the animals’ behavior. Numbers in parentheses refer to the number of free parameters in the model (see also Online Resources 2 (model equations), 4 (table of model selection), and 5 (model selection per inactivation condition)). b The “winning” model was a Rescorla-Wagner model (RW3), in which the animals track the value of both nose pokes over an extended history of outcomes by learning from reward and punishment (i.e., reward versus reward omission)

Fig. 3

Model coefficients. Best-fit model parameters for each session. Inactivation of the PrL and lOFC impaired reward and punishment learning, whereas inactivation of the IL and mOFC impaired punishment learning and reduced choice perseveration (i.e., repeated choices for the same nose poke hole). ACC, n = 10 rats; PrL, n = 12 rats; IL, n = 9; mOFC, n = 9; lOFC, n = 9. *P < 0.05, **P < 0.01, ***P < 0.001 (post hoc Holm-Sidak test; see also Online Resource 1)

Fig. 4

Visual summary. PFC subregions have distinct, albeit overlapping, functions in value-based behaviors. All regions except the ACC are involved in punishment learning. Shown is Z-score of B/M effect ((mean_BM − mean_Sal)/SD_sal)

Fig. 5

Simulated data. We simulated probabilistic reversal learning sessions (50 simulations per condition (i.e., per pixel), 200 trials per session), to assess how changes in the computational model parameters affect conventional measures of task performance in the (simulated) data. Explore/exploit parameter β was fixed at the animals’ grand average from the experimental data (β = 1.686). The highest number of reversals can be obtained by a combination of high learning and high stickiness, whereas the number of rewarded trials could be maximized by having high learning and an intermediate value of the stickiness parameter. Win-stay/lose-switch measures were mostly dependent on the stickiness parameter, but can be modulated by both reward and punishment learning. Thus, whether changes in the computational model parameters lead to significant changes in conventional task measures is highly dependent on the baseline behavior of the animal and size and direction of the effects.