Signals in human striatum are appropriate for policy update rather than value prediction

Abstract

Influential reinforcement learning theories propose that prediction error signals in the brain's nigrostriatal system guide learning for trial-and-error decision-making. However, since different decision variables can be learned from quantitatively similar error signals, a critical question is: what is the content of decision representations trained by the error signals? We used fMRI to monitor neural activity in a two-armed bandit counterfactual decision task that provided human subjects with information about forgone and obtained monetary outcomes so as to dissociate teaching signals that update expected values for each action, versus signals that train relative preferences between actions (a policy). The reward probabilities of both choices varied independently from each other. This specific design allowed us to test whether subjects' choice behavior was guided by policy-based methods, which directly map states to advantageous actions, or value-based methods such as Q-learning, where choice policies are instead generated by learning an intermediate representation (reward expectancy). Behaviorally, we found human participants' choices were significantly influenced by obtained as well as forgone rewards from the previous trial. We also found subjects' blood oxygen level-dependent responses in striatum were modulated in opposite directions by the experienced and forgone rewards but not by reward expectancy. This neural pattern, as well as subjects' choice behavior, is consistent with a teaching signal for developing habits or relative action preferences, rather than prediction errors for updating separate action values.

Figure 1.

Experimental design. A, Timeline of a single trial. B, Example from one subject of changing reward probabilities for both slots. ITI, Intertrial interval.

Figure 2.

A, Neural correlates of R_c − κR_u. B, C, Overlapping view of the neural correlates of prediction errors of chosen choices (B, δ_chosen and −δ_chosen) and forgone choices (C, δ_unchosen and −δ_unchosen). p < 0.05.

Figure 3.

A, Effect of decision variables in striatum. BOLD activity positively correlates with R_c and −R_u (p < 0.05) but not with −Q_c or Q_u, even at a loose threshold [p = 0.01, uncorrected (unc)]. The dotted boxes surround the pairs of effects expected to be significant for value-based learning [R_c − Q_c and −(R_u − Q_u)] and the solid box surrounds those for policy learning (R_c − κR_u). B, Similar activities (bilateral insula, anterior cingulate cortex, and dorsolateral prefrontal cortex) were positively correlated with −R_c and R_u (p < 0.05).

Figure 4.

A, Error signaling ROI in left ventral striatum, identified from the conjunction of R_c and −R_u across subjects [p < 0.001, uncorrected (unc)]. B, In left striatum (A, circles), the neural effect size for −R_u was positively correlated, across subjects, with the weight for the unchosen reward, κ, estimated from choice behavior (p = 0.018, Bonferroni corrected; r = 0.57).