Neurophysiology of Reward-Guided Behavior: Correlates Related to Predictions, Value, Motivation, Errors, Attention, and Action

Abstract

Many brain areas are activated by the possibility and receipt of reward. Are all of these brain areas reporting the same information about reward? Or are these signals related to other functions that accompany reward-guided learning and decision-making? Through carefully controlled behavioral studies, it has been shown that reward-related activity can represent reward expectations related to future outcomes, errors in those expectations, motivation, and signals related to goal- and habit-driven behaviors. These dissociations have been accomplished by manipulating the predictability of positively and negatively valued events. Here, we review single neuron recordings in behaving animals that have addressed this issue. We describe data showing that several brain areas, including orbitofrontal cortex, anterior cingulate, and basolateral amygdala signal reward prediction. In addition, anterior cingulate, basolateral amygdala, and dopamine neurons also signal errors in reward prediction, but in different ways. For these areas, we will describe how unexpected manipulations of positive and negative value can dissociate signed from unsigned reward prediction errors. All of these signals feed into striatum to modify signals that motivate behavior in ventral striatum and guide responding via associative encoding in dorsolateral striatum.

Keywords: Attention; Decision-making; Motivation; Prediction error; Reward; Value.

Fig. 1

a Circuit diagram demonstrating connectivity between brain regions involved in reward-guided decision-making. Arrows represent direction of information flow where single-headed arrows are unidirectional and double-headed arrows are reciprocal. Words in shaded box specify functions and shading of box provides a general idea of the role that nearby anatomical labels play in the strength of these functions. b Interplay of functions related to reward-guided decision-making. Orbitofrontal cortex OFC, dorsal-lateral prefrontal cortex PFC, basolateral amygdala ABL, anterior cingulate cortex ACC, parietal cortex Parietal, premotor cortex PM, nucleus accumbens NA, dorsal-medial striatum DMS, dorsolateral striatum DLS, ventral tegmental area VTA, dopamine DA, substantia nigra compacta SNc, globus pallidus GP, thalamus Thal, substantia nigra reticulata SNr, prediction error PE. Adapted from Bissonette et al. (2014) and Burton et al. (2014)

Fig. 2

a Tasks that dissociate value from motivation/salience correlate by varying appetitive and aversive outcomes. In these tasks, one trial type promises a large reward with little punishment (app = appetitive); another promises a small reward with little punishment (neu = neutral); and a third promises the small reward, but threatens the animal with a large punishment (aver = aversive). Both primates and rats performing tasks like these prefer large reward and dislike large punishment (aver) relative to neutral, but are highly motivated by both as indicated by faster reaction times and better performance. Thus, theoretically, if neurons in the brain participate in value computations, then they should fire highest for appetitive trial types and lowest for aversive trial types (“value”). If neurons participate in signals that reflect motivation/salience, their activity should be high for both appetitive and aversive trial types. b Monkey study that dissociated value from motivation. Accuracy and RT data from primates, illustrating that monkeys were faster and more accurate for large reward and punishment trials, versus neutral trials. c Neural recordings in primate OFC demonstrate higher activity for large over small reward cues, whereas activity in premotor cortex (PM) d reflected the level of motivation associated with those outcomes. e–g, behavior data from rats performing a similar task. Rats licked more for large, appetitive outcomes, and less for punishing trials and were more accurate f and faster g for large rewards and potential punishing trials compared to neutral trials. h Neural recordings in nucleus accumbens show higher activity for high-valued cues than for lower-valued cues in one neural population, while other NAc neurons i showed salience signals for both high-valued reward and possible punishment trial types. Firing rates were normalized by subtracting the baseline and dividing by the standard deviation. Ribbons represent standard error of the mean (SEM). Gray-dashed aversive (aver); Black appetitive (app); Gray solid neutral (Neu). Adapted from Bissonette et al. (2013)

Fig. 3

Neural activity in VTA and ABL responses to unexpected reward and omission consistent with Rescorla–Wagner or Pearce–Hall attention models, respectively. a Example trial types representing up-shifts in value, with an unexpected increase in reward quantity (left) or unexpected decrease in wait time for reward (right). Deflections reflect time of events. Heavy black lines and fluid drops reflect unexpected reward delivery (i.e., up-shift). In this task, odors predicted short (0.5 s) or long (1–7 s) delays to reward during “delay” blocks. In “size” blocks, odors predicted large (2 boli) or small (1 bolus) reward. b Example trial types representing a down-shifting in value, with an unexpected decrease in reward quantity or increase in wait time for reward. Deflections reflect time of events. Dashed gray lines and fluid drops reflect unexpected reward omission (i.e., down-shift). c, d Signals predicted by the Rescorla–Wagner (c) and Pearce–Hall (d) models after unexpected delivery (black) and omission (gray) of reward. e, f Average firing during the 500 ms after reward delivery in dopamine neurons in VTA (e) and for ABL (f) during the first ten trials when value of delivered reward was unexpectedly higher (up-shifts = black) and in blocks when the value of the reward was unexpectedly lower (down-shifts = gray) normalized to the maximum firing rate. Error bars indicate SEMs. Adapted from Roesch et al. (2010a)

Fig. 4

Neural correlates across primate striatum. a Visually guided saccade task with an asymmetric reward schedule. After the monkey fixated on the FP (fixation point) for 1200 ms, the FP disappeared and a target cue appeared immediately on either the left or right, to which the monkey made a saccade to receive a liquid reward. The dotted circles indicate the direction of gaze. In a block of 20–28 trials (e.g., left-big block), one target position (e.g., left) was associated with a big reward and the other position (e.g., right) was associated with a small reward. The position–reward contingency was then reversed (e.g., right-big block). b Subdivisions of the primate striatum. c Percentage of neurons that showed large-reward preference for each subdivision of striatum. Modified from Nakamura et al. (2012), Roesch et al. (2009), Burton et al. (2014), Roesch and Bryden (2011)

Fig. 5

Neural correlates across rat NAc and DLS. Odor-guided choice task during which the delay to and size of reward were independently varied in ~60 trial blocks (i.e., “blocks 1–4”). Upon illumination of house lights, rats started the trial by poking into the central port. After 500 ms, an odor signaled the trial type. For odors 1 and 2, rats had to go to a left or right fluid well to receive reward (forced-choice trials). A third odor signaled that the rat was free to choose either well to receive the reward that was associated with that response direction during the given block of trials. In blocks 1–4, the length of the delay (blocks 1 and 2) to reward and the size of reward (blocks 3 and 4) were manipulated: short delay = 0.5 s wait before delivery of 1 bolus reward; Long delay = 1–7 s wait before 1 bolus reward; Big reward = 0.5 s wait for 2–3 boli reward; Small reward = 0.5 s wait before 1 bolus reward. Throughout each recording session, each of these trial types were associated with both directions and all three odors, allowing us to examine different associative correlates. b Locations of recording sites in rat NAc, DMS, and DLS and percentage of significantly modulated neurons. More NAc neurons encoded high-valued options (NAc, black bar). Representations of outcome were evenly distributed in DLS, while number of response direction encoding neurons was significantly elevated (contralateral gray bar). Chi-square was used to compare counts of neurons. Hi High value; Lo Low value; Con contralateral to recording site; Ipsi Ipsilateral to the recording site. c and d Example of a single neuron recorded in DLS for each of the trial types for forced (c) and free (d) choice odors. Modified from Nakamura et al. (2012), Roesch et al. (2009), Burton et al. (2014), Roesch and Bryden (2011)