Transient activation of midbrain dopamine neurons by reward risk

Abstract

Dopamine neurons of the ventral midbrain are activated transiently following stimuli that predict future reward. This response has been shown to signal the expected value of future reward, and there is strong evidence that it drives positive reinforcement of stimuli and actions associated with reward in accord with reinforcement learning models. Behavior is also influenced by reward uncertainty, or risk, but it is not known whether the transient response of dopamine neurons is sensitive to reward risk. To investigate this, monkeys were trained to associate distinct visual stimuli with certain or uncertain volumes of juice of nearly the same expected value. In a choice task, monkeys preferred the stimulus predicting an uncertain (risky) reward outcome. In a Pavlovian task, in which the neuronal responses to each stimulus could be measured in isolation, it was found that dopamine neurons were more strongly activated by the stimulus associated with reward risk. Given extensive evidence that dopamine drives reinforcement, these results strongly suggest that dopamine neurons can reinforce risk-seeking behavior (gambling), at least under certain conditions. Risk-seeking behavior has the virtue of promoting exploration and learning, and these results support the hypothesis that dopamine neurons represent the value of exploration.

Figure 1

Risk preference in a choice task. A, Two images were presented simultaneously. Fixation of gaze on stimulus C for 0.5 s resulted in certain delivery of 125 μL of juice after a further delay of 1.0 s, whereas fixation on stimulus U resulted in 240 μL of juice on a pseudorandomly selected 50% of trials, and no juice on the remaining trials. These two images were used in all 3 monkeys, but were reversed in meaning between monkeys O and F (the case of monkey F is illustrated here) Distinct and novel stimuli were used in some cases, with similar results (not shown). B, Choice behavior in a single session of 1000 trials that lasted for 70 minutes. Each vertical line represents a single trial, starting with the first trial in the top left corner and ending with the last trial in the bottom right corner. Choices for stimulus C are in blue. Choices for stimulus U that resulted in large reward outcome are in red, and those that resulted in no reward are in green. C, Fraction of choices for stimulus U over time during the same session illustrated in B. The thick line represents bins of 50 trials, whereas the thin line represents bins of 10 trials. D, Percentage of choices for stimulus U by day. Day 1 is the first day on which choice behavior was measured after the start of Pavlovian conditioning. 70 to 1000 trials were performed each day. The lighter area of each bar represents ± 2 standard deviations from the mean of the binomial distribution with p = 0.5 and `n' equal to the total number of choice trials performed. Monkey O in black, monkey F in red, and monkey L in blue. The data shown in B and C corresponds to the second day after the start of Pavlovian conditioning in monkey F.

Figure 2

Behavioral preference for stimulus U during Pavlovian conditioning. A, Animals viewed stimulus U on a higher percentage of trials than stimulus C. Eye position was measured every 5 ms, and the fraction of trials (mean±sem) for each 5 ms bin in which gaze was within ±2 degrees from the center of the image was calculated for about 5000 trials of each type (collected across all sessions in which a neuron was recorded. Stimulus onset was at time “0.” B, Preference for stimulus U over C persisted for multiple sessions within a day and across days. Each point represents the preference for stimulus U in a single session of about 60 trials of each type. The sessions shown are only those in which a dopamine neuron was simultaneously recorded (1–6 dopamine neurons per day). The preference was calculated as the average percentage of trials spent viewing stimulus U in the 0.5 s period before reward outcome (as shown in A) minus the average percentage of trials spent viewing stimulus C. In the first sessions of a day, monkeys usually viewed each stimulus on every trial, producing a “ceiling effect” that may have obscured any preference. No clear relationship was observed between this measure of viewing preference and neuronal preference (not shown).

Figure 3

Positions of recorded neurons in relation to neuronal risk preference. Monkey F in red, monkey O in black. A. Estimated positions of neurons in drawings of coronal sections (adapted from Paxinos et al., 2000) at interaural 7.05 and 8.85 mm anterior of the interaural line. Triangles indicate neurons that displayed statistically significant discrimination of stimulus U versus stimulus C. The blue arrow indicates the anatomical location of the neuron for which data is displayed in figure 4. All neurons from each hemisphere are shown as though in one hemisphere. The more anterior section (8.85 mm) displays all neurons recorded between 8.0 and 9.5 mm anterior of the interaural line, whereas the posterior section (7.05 mm) includes all neurons recorded between 5.5 and 7.5 mm. According to the classification scheme depicted here, dopamine neurons would be expected to be found within VTA, parabrachial plexus (PBP), and paranigral nucleus (PN) (all classified as “A10” dopamine neurons), as well SN pars compacta (SNc) (A9). Some of the neurons recorded at more posterior levels (less than approximately 6 mm anterior to the interaual line) may have been in the retrorubral field (A8) (not shown here). Positions of neurons were estimated as described in Experimental Procedures. The fact that some neurons appear in this figure to lie outside of dopaminergic regions is most likely due to errors in measuring the positions of neurons and in translating those positions to the atlas. The errors are likely to be greater in the dorsal-ventral than medial-lateral dimension. B. The position of electrode tracks lateral of the midline and anterior of the interaural line. Several neurons were typically recorded on a single track in a single day (Fig. 2B). C. Neuronal preferences for stimulus U over C as a function of neuronal position lateral from the midline (without regard to hemisphere) (left), anterior of the interaural line (middle), and dorsal of the interaural line (right). Neuronal preference was calculated as the percentage by which the response (firing rate) to stimulus U was greater than stimulus C (firing rate following onset of stimulus U minus stimulus C, divided by firing rate to stimulus C and multiplied by 100). Note that some neurons found to have very large preferences also had very low firing rates (spontaneously and in response to stimulus C; see Fig. 5B).

Figure 4

Responses to stimuli U and C within a single dopamine neuron. Top, peri-stimulus time histograms, demonstrating a stronger mean response to stimulus U than to stimulus C. Bin size = 50 ms. Bottom two panels, rasters of 60 trials each of stimulus U (middle) and stimulus C (bottom), arranged in each panel in chronological order from top to bottom.

Figure 5

Dopamine neurons are more strongly activated by a conditioned stimulus (“U”) that predicts an uncertain reward outcome than by a stimulus (“C”) that predicts a certain reward outcome. A, Peri-stimulus time histograms of firing rate for the entire population of individually recorded dopamine neurons (mean ± s.e.m.) following onset of conditioned stimuli at time “0” (stimulus U in red, stimulus C in black; 33 neurons in monkey O at left, 41 neurons in monkey F at right). To further characterize responses, firing rate was measured in each neuron in a window of 120 – 400 ms following stimulus onset, as indicated by the bars shown beneath the histograms. B, Scatter plot of firing rates following stimuli U and C. Each point represents a single neuron. The 20 points in red represent neurons in which responses to stimulus U were significantly greater that to stimulus C (p<0.05, unpaired t-tests across trials).

Figure 6

The neuronal response to stimulus U did not depend on the reward outcome of the previous trial. Responses to stimulus U were segregated into two groups depending on whether or not the most recent presentation of stimulus U was or was not followed by juice reward. Left, peri-stimulus time histograms (stimulus U in red, stimulus C in black). Stimulus onset occurred at time “zero.” Right, scatter plot of responses in individual neurons. The 5 points in red signify neurons in which responses were significantly larger, or smaller, depending on the outcome of the previous trial (without correction for multiple comparisons).

Figure 7

Average neuronal responses to reward outcomes. Juice onset (or offset of stimulus U on trials in which juice was omitted) occurred at time “zero.” Consistent with previous studies, when the delivery of juice is uncertain following stimulus U, its delivery causes a strong activation (red), whereas its omission causes a suppression of firing rate below the baseline level (blue). When the delivery of juice is nearly certain following stimulus C, its delivery causes only a small activation (black). The black horizontal line between 80 and 300 ms indicates the period in which firing rates were measured for comparison, as described in the results.