Moment-to-moment tracking of state value in the amygdala

Abstract

As an organism interacts with the world, how good or bad things are at the moment, the value of the current state of the organism, is an important parameter that is likely to be encoded in the brain. As the environment changes and new stimuli appear, estimates of state value must be updated to support appropriate responses and learning. Indeed, many models of reinforcement learning posit representations of state value. We examined how the brain mediates this process by recording amygdala neural activity while monkeys performed a trace-conditioning task requiring fixation. The presentation of different stimuli induced state transitions; these stimuli included unconditioned stimuli (USs) (liquid rewards and aversive air puffs), newly learned reinforcement-predictive visual stimuli [conditioned stimuli (CSs)], and familiar stimuli long associated with reinforcement [fixation point (FP)]. The FP had a positive value to monkeys, because they chose to foveate it to initiate trials. Different populations of amygdala neurons tracked the positive or negative value of the current state, regardless of whether state transitions were caused by the FP, CSs, or USs. Positive value-coding neurons increased their firing during the fixation interval and fired more strongly after rewarded CSs and rewards than after punished CSs and air puffs. Negative value-coding neurons did the opposite, decreasing their firing during the fixation interval and firing more strongly after punished CSs and air puffs than after rewarded CSs and rewards. This representation of state value could underlie how the amygdala helps coordinate cognitive, emotional, and behavioral responses depending on the value of one's state.

Figure 1.

Experimental paradigm and monkey behavior. A, Trace-conditioning task. Sequence of events for the strong positive, weak positive, and negative trial types. After monkeys learn the initial CS–US relationships, positive and negative trial types are reversed, such that the strong positive image becomes negative and the negative image becomes strongly positive. Weak positive images do not undergo reversal. Reinforcement occurs with a 80% probability on all trial types. B, C, Mean probability of a response across experiments, for anticipatory licking (B) and blinking (C), before reinforcement delivery as a function of time within trials.

Figure 2.

Amygdala neurons encode the value of states initiated by FP presentation. A, B, PSTHs showing two example amygdala neurons, one encoding positive CS value and increasing firing rate during fixation point presentation (A), and the other encoding negative CS value and exhibiting a decrease in firing to the fixation point (B). C, Population average of responses to fixation point for neurons encoding positive or negative CS value and for neurons not encoding CS value. Shaded areas indicate SEM. D, Bar chart showing the percentage of cells with increases (blue), decreases (red), and no change (black) in response level during fixation point presentation, for positive, negative, and non-value-coding neurons. The number of cells of each type is indicated. The majority of positive value-coding cells increases firing to the FP, and the majority of negative value-coding cells decreases firing to the FP. Non-value-coding cells have no discernable pattern in firing to the FP, with increases, decreases, and no change in responses to the FP occurring with approximately equal frequency.

Figure 3.

Neural responses during the fixation interval are not related to reinforcement uncertainty. A, B, Cumulative distributions showing the proportion of positive (A) and negative (B) value-coding neurons as a function of the difference in firing rate between the fixation interval and the CS or trace interval for rewarded (blue) and punished (red) trials (FP interval activity − CS/trace interval activity). For this comparison, for each cell, we used either the firing rate from 90–390 ms after CS onset (CS interval) or from 390–1800 ms after CS onset (trace interval), depending on which interval the cell encoded value for more strongly (see Materials and Methods). Distribution means were tested for significant difference from 0 using a one-tailed t test.

Figure 4.

Amygdala neurons encode the value of states initiated by CS presentation. A–D, PSTHs for four example amygdala neurons that respond in a graded manner during the visual stimulus (A, B) or trace (C, D) intervals. Example cells 3 and 5 (A, C) encode positive value, and cells 4 and 6 (B, D) encode negative value. In each case, responses to the CS are correlated with the value of the associated US, consistent with the encoding state value of the neurons. E, Scatter plot of positive and negative NDIs, computed using ROC analyses that compared activity on small reward trials with large reward trials (positive NDI) or punished trials (negative NDI). Each data point represents a single experiment. Blue data points, positive value-coding cells; red data points, negative value-coding cells.

Figure 5.

A–C, Value encoding is widespread within the amygdala. MRI reconstruction (A–C, posterior to anterior) of recording locations with sites labeled according to whether positive and negative NDIs are significantly different from 0.5. Images were acquired with a two-dimensional inversion recovery sequence, with 2-mm-thick slices. Consequently, recording sites spanning 2 mm in the anteroposterior dimension were collapsed onto single images; therefore, in many cases, recording sites appear as if they overlapped in the figure. Black filled circles, Non-value-encoding; open colored circles, value-encoding, one NDI significantly different from 0.5; filled colored circles, both NDIs significantly different from 0.5; red symbols, negative value-encoding; green symbols, positive value-encoding. Value-coding neurons were recorded from areas that likely encompass both the central and the basolateral collection of nuclei. D, Dorsal; L, lateral; M, medial; V, ventral.

Figure 6.

Amygdala neurons appear to track state values across the CS and US intervals. A, B, Positive (A) and negative (B) US NDI plotted as a function of positive (A) and negative (B) CS NDI. Red lines, Linear regressions fit to the data.

Figure 7.

Monkeys' differential behavior is correlated with neuronal responses. A–D, Positive (A, B) and negative (C, D) BDIs based on licking responses plotted against positive (A, B) and negative (C, D) CS discrimination indices for neurons (NDIs). Separate plots shown for positive (A, C) and negative (B, D) value-coding neurons. E, F, Negative BDIs based on blinking data plotted against negative CS NDIs, with separate plots shown for positive (E) and negative (F) value-coding cells. Positive BDIs for blinking are not shown because blinking on rewarded trials was rare. Black lines, Linear regressions fit to the data.