Planning activity for internally generated reward goals in monkey amygdala neurons

Abstract

The best rewards are often distant and can only be achieved by planning and decision-making over several steps. We designed a multi-step choice task in which monkeys followed internal plans to save rewards toward self-defined goals. During this self-controlled behavior, amygdala neurons showed future-oriented activity that reflected the animal's plan to obtain specific rewards several trials ahead. This prospective activity encoded crucial components of the animal's plan, including value and length of the planned choice sequence. It began on initial trials when a plan would be formed, reappeared step by step until reward receipt, and readily updated with a new sequence. It predicted performance, including errors, and typically disappeared during instructed behavior. Such prospective activity could underlie the formation and pursuit of internal plans characteristic of goal-directed behavior. The existence of neuronal planning activity in the amygdala suggests that this structure is important in guiding behavior toward internally generated, distant goals.

Figure 1

Reward-saving behavior in monkeys. (a) Sequential saving task. Animals chose freely to save or spend reward and determined internally the length of each saving sequence. Consecutive save choices increased reward amounts (determined by interest rate); spend choice resulted in reward delivery. Sequences lasted up to 9 consecutive trials (~12 s cycle time/trial). (b) Saving behavior, reward increases, and subjective value functions for different interest rates. Bars: relative frequencies with which animals produced different sequences, combined across animals. Green curves: reward amounts for different sequences. Magenta: subjective values (normalized), combining choice frequencies with reward magnitudes. With highest interest rate, reward stagnated after seven trials; most neuronal recordings involved intermediate interest rates. (c) Monkeys adapted their saving behavior to interest rate. Linear regression of weighted mean sequence length on interest for main task (black, n = 17) and control test with uncued changes in interest (magenta, n = 9). Data combined across animals. (d) Linear regression of reaction time on final sequence length. Reaction times (equally populated bins pooled over animals and interest rates, z-normalized within sessions) on spend trials (black, averaged over n = 3,033 trials) and save trials (magenta, averaged over n = 8,500 trials) were shorter for longer sequences (i.e. higher rewards). (e) Logistic regression of trial-by-trial choices. Spend/save value: subjective value associated with spending/saving on current trial; sequence value: subjective sequence value (spend value on final trial). Bias: constant; Cue position: left/right save cue position; Juice/day: consumed juice; Monkey: animal identity. **P < 0.005, *P < 0.05; n.s. not significant. Error bars: s.e.m.

Figure 2

A single amygdala neuron with prospective activity that reflected the value of the monkey’s internal saving plan. (a) Activity during step-by-step saving depended on the final saving sequence that the animal eventually produced. Specifically, activity depended on the subjective value of the current sequence (‘sequence value’), which would only be achieved several trials ahead. Upper panels: activity (spike density functions) during three saving sequences of different lengths. Activity during fixation (yellow area) was highest for the sequence in which the monkey would eventually spend on the fifth trial, as this sequence had the highest subjective value (Imp/s: impulses per second; raster display: ticks indicate impulses, rows indicate trials). Lower panel: activity averages for all sequence lengths (e.g. light-pink activation indicates mean fixation activity for all five-trial sequences, averaged over trials one to five). Activity reflected sequence value (magenta curve, normalized), rather than linear sequence length or objective reward amount (green curve, normalized). Behaviorally derived sequence values reflected the animal’s preferences for different combinations of sequence length and final reward—five-trial sequences had the highest value as the monkey chose them most frequently. Saving sequences were freely determined by the animal; visual stimulation was constant across sequences. (b) Within-trial activity sorted according to sequence value (terciles). (c) Linear regression of activity on sequence value. Different value levels resulted from different sequence lengths as shown in (a). (d) Multiple regression coefficients (betas ± s.e.m., Eq. 6). (e) Activity in the imperative task, when saving was instructed, did not reflect sequence value.

Figure 3

Different forms of planning activity in four single amygdala neurons. (a) Activity of this neuron, as in Fig. 2, reflected sequence value across all trials. Right panel: regression betas obtained by fitting Eq. 6 to neuronal activity. (b) Activity of this neuron at trial start before fixation (“Pre-fix period”) reflected sequence value specifically on the first trial of each sequence (bold colors), but not on subsequent trials (light colors). Right panel: regression betas obtained by fitting Eq. 8 to neuronal activity. First trial indicator: indicator variable for the first trial in a saving sequence. First trial indicator × sequence value: regressor for testing sequence value coding specifically on first saving trials. (Statistics for first-trial effects were based on data from all save trials including bold and light colored data. The linear regression in the middle panel remained significant when the effect of outliers was reduced using robust regression.) (c) Activity of this neuron during the fixation and cue periods reflected final sequence length, rather than sequence value, across all saving trials. Activity was higher for shorter sequences. Right panel: regression betas obtained by fitting Eq. 7 to neuronal activity. (d) Activity of this neuron in the fixation period reflected sequence length specifically on first saving trials. Right panel: regression betas obtained by fitting Eq. 9 to neuronal activity. First trial indicator × sequence length: regressor for testing sequence length coding specifically on first saving trials.

Figure 4

Planning activity in amygdala neurons: population data. (a,b) Planning activity (z-normalized) of 72 neurons encoding sequence value across all trials or specifically on first trials. (b) Population activity (magenta, n = 93 responses) reflected sequence value (r² = 0.91, P < 0.0001, linear regression, n = 7) rather than sequence length (r² = 0.38, P > 0.1). (c,b) Planning activity of 71 neurons encoding sequence length across all trials or specifically on first trials. (d) Population activity (magenta, n = 92 responses) reflected sequence length (r² = 0.85, P = 0.0035, n = 7) rather than sequence value (r²= 0.14, P > 0.4). (e, f) Activity of neurons tested in the imperative task failed to reflect sequence value or sequence length when saving was instructed (data from 30 neurons encoding sequence value and 29 neurons encoding sequence length). (g) Regression betas for observed data (orange, n = 829 responses from 329 neurons, collapsed across sequence value and sequence length) and trial-shuffled data (black, scaled down 1,000 times). The distribution of observed data was shifted towards higher positive and negative values (Kolmogorov-Smirnov test). (h) Histological reconstruction of 72 sequence value neurons and 71 sequence length neurons. Green, white, pink, yellow and blue symbols: example neurons in Fig. 2 and Fig. 3a–d, respectively. Collapsing across anterior-posterior dimension resulted in symbol overlap. (i) Proportion of neurons with planning activity (n = 123 neurons, collapsed across sequence value and sequence length) in basolateral and centromedial amygdala (P = 0.005, χ²-test) and corresponding recording depths (reference: bregma).

Figure 5

Adaptation dynamics of planning activity, reward proximity control. (a) Sequence-by-sequence adaptation in a single neuron encoding sequence length. Activity changes from spend to save trials (dashed lines) reflected changes in sequence length between successive sequences. Gray curves: sequence-averaged activity (thick line) and trial-by-trial activity (thin line). Green curve: sequence length. Blue curve: within-sequence reward proximity. Arrows: examples for activity changes scaling with sequence length changes. Colored boxes indicate sequences and corresponding lengths. (b) Linear regression of activity of the neuron in (a) on sequence length (left, n = 41), difference in length between subsequent sequences (ΔSequence length, middle, n = 7), and reward proximity (right, n = 41). (c) Population data. Left: sequence value responses (n = 61); activity changes at sequence transitions reflected changes in sequence value (linear regression). Middle: sequence length responses (n = 55); activity changes reflected changes in sequence length. Right: Population activity (sequence value and sequence length responses, n = 116) was unrelated to within-sequence reward proximity. (d) Regression betas for planning activity and reward proximity (n = 116 sequence value and sequence length responses, Kolmogorov-Smirnov test). (e) Behavioral-neuronal adaptation in sequence value neurons. Upper: With a new testing session, planning activity adapted readily to current interest rate, instep with behavior (r = 0.82, P = 1.7 × 10⁻⁴; both Medians = 1, n = 61). Lower: Neurons typically reached adaptation criterion within the first sequence (Median = −3, implying adaption within 3 trials before end of first sequence, t₆₀ = −10.17, P = 1.0 × 10⁻¹⁴, one-sample t-test).

Figure 6

Relationship between amygdala planning activity and behavioral performance. (a) Relationship to saving efficiency. Stronger planning activity (sign-corrected regression betas, collapsed across responses encoding sequence value or sequence length across all trials, n = 116) predicted behavioral saving efficiency (accumulated sequence value per unit time, normalized, linear regression). This effect was confirmed in a partial correlation analysis (P < 0.001) that factored out potential confounding variables. (b) Relationship to performance errors. Bars show regression betas (± s.e.m) from a population analysis (combining sequence value and sequence length responses, n = 116) for trials immediately preceding errors (Pre-), error trials (Error), and trials following errors (Post-). The relationship between activity and planning variables was significantly reduced on error trials, when the animals failed to progress towards their saving goal (t₁₄₅₃ = −2.69, P < 0.01, dependent-samples t-test comparing betas on pre-error and error trials), and subsequently reappeared after error correction (t₁₄₅₃ = 3.47, P < 0.001, dependent-samples t-test comparing betas on error and post-error trials).