A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning

PLoS Comput Biol. 2018 Jan 4;14(1):e1005925. doi: 10.1371/journal.pcbi.1005925. eCollection 2018 Jan.

Abstract

Reinforcement learning has been widely used in explaining animal behavior. In reinforcement learning, the agent learns the value of the states in the task, collectively constituting the task state space, and uses the knowledge to choose actions and acquire desired outcomes. It has been proposed that the orbitofrontal cortex (OFC) encodes the task state space during reinforcement learning. However, it is not well understood how the OFC acquires and stores task state information. Here, we propose a neural network model based on reservoir computing. Reservoir networks exhibit heterogeneous and dynamic activity patterns that are suitable to encode task states. The information can be extracted by a linear readout trained with reinforcement learning. We demonstrate how the network acquires and stores task structures. The network exhibits reinforcement learning behavior and its aspects resemble experimental findings of the OFC. Our study provides a theoretical explanation of how the OFC may contribute to reinforcement learning and a new approach to understanding the neural mechanism underlying reinforcement learning.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Animals
  • Cognition
  • Decision Making*
  • Electrophysiology
  • Frontal Lobe
  • Humans
  • Learning*
  • Magnetic Resonance Imaging
  • Markov Chains
  • Models, Neurological
  • Nerve Net*
  • Neural Networks, Computer*
  • Neurons / physiology*
  • Prefrontal Cortex / physiology*
  • Probability
  • Reinforcement, Psychology*
  • Spatial Learning / physiology*

Grants and funding

This work is supported by the Chinese Academy of Sciences (http://english.cas.cn/) Hundreds of Talents Program and Science and Technology Commission of Shanghai Municipality (http://www.stcsm.gov.cn/english/) (15JC1400104) to TY, and by Public Projects of Zhejiang Province (http://www.zjzwfw.gov.cn/) (2016C31G2020069) and the 3rd Level in Zhejiang Province (http://www.zjzwfw.gov.cn/) "151 talents project” to ZC. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.