When Does Model-Based Control Pay Off?

PLoS Comput Biol. 2016 Aug 26;12(8):e1005090. doi: 10.1371/journal.pcbi.1005090. eCollection 2016 Aug.


Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to "model-free" and "model-based" strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

MeSH terms

  • Adolescent
  • Adult
  • Aged
  • Algorithms
  • Computational Biology
  • Decision Making / physiology*
  • Female
  • Humans
  • Learning / physiology*
  • Male
  • Middle Aged
  • Models, Theoretical*
  • Reinforcement, Psychology
  • Task Performance and Analysis
  • Young Adult

Grants and funding

This research was supported by grant N00014-14-1-0800 from the Office of Naval Research (http://www.onr.navy.mil/) and and based upon work supported by the Center for Brains, Minds and Machines (CBMM, https://cbmm.mit.edu/), funded by NSF STC award CCF-1231216. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.