Neural computations underlying action-based decision making in the human brain

Abstract

Action-based decision making involves choices between different physical actions to obtain rewards. To make such decisions the brain needs to assign a value to each action and then compare them to make a choice. Using fMRI in human subjects, we found evidence for action-value signals in supplementary motor cortex. Separate brain regions, most prominently ventromedial prefrontal cortex, were involved in encoding the expected value of the action that was ultimately taken. These findings differentiate two main forms of value signals in the human brain: those relating to the value of each available action, likely reflecting signals that are a precursor of choice, and those corresponding to the expected value of the action that is subsequently chosen, and therefore reflecting the consequence of the decision process. Furthermore, we also found signals in the dorsomedial frontal cortex that resemble the output of a decision comparator, which implicates this region in the computation of the decision itself.

Fig. 1.

Experimental Design and Behavior. (A) Subjects were presented with a choice cue after which they had to respond within 2.5 s by performing a saccade to the red target circle or a right handed button press. Once a response was registered the screen was immediately cleared for a short delay and subsequently the outcome was revealed (6 s after trial onset) indicating either receipt of reward or no reward. Inter-trial-intervals varied between 1 and 8 s. (B) Example reward probabilities for saccades and button presses as a function of the trial number. The probability of being rewarded following choice of either the hand or eye movement was varied across the experiment independently for each movement. (C) Fitted model choice probability (red) and actual choice behavior (blue) shown for a single subject. (D) Actual choice behavior versus model predicted choice probability. Data are pooled across subjects, the regression slope is shown as a line, vertical bars, SEM.

Fig. 2.

Action values. (A) Region of supplementary motor area showing correlations with action values for hand movement (Vh/green) and a region of pre-SEF showing correlations with action-values for eye movements (Ve/red). T-maps are shown from a whole brain analysis thresholded at P < 0.001 uncorrected (see Fig. S1 for a version with color bars relating to t stats). (B) Average effect sizes of Ve (red) and Vh (green) extracted from SEF and SMA. The effects shown here were calculated from trials independent of those used to functionally identify the ROI. Note that only Ve but not Vh modulate the signal in preSEF, and that activity in SMA shows the opposite pattern. Vertical lines, SEM.

Fig. 3.

Chosen values. (A) Brain regions showing significant correlations with the value of the action chosen. Areas shown include vmPFC, intra-parietal sulcus and posterior cingulate cortex. Threshold is set at P < 0.001. (B) Distinct forms of the value chosen signal are present within vmPFC. The area depicted in yellow indicates voxels that correlate with the value of the chosen action irrespective of whether the action taken is a hand or an eye movement. The area depicted in green correlates only with the value chosen on trials when the hand movement is chosen but not when the eye movement is chosen. Finally the area depicted in red indicates voxels correlating with value chosen only on trials when the eye movement is selected but not the hand movement. The results suggest an anterior to posterior trend in the selectivity of voxels to these different types of value chosen signals. Bar plots show effect sizes averaged across subjects for the action specific value chosen signals in the three areas (left: red area, middle: green area, right: yellow area). Bars shown in chromatic color are significantly different from zero (t test, P < 0.05). Similar to bar plots in Fig. 2B, effects were calculated from a data sample independent of the one used to functionally identify the ROI. Vertical lines, SEM.

Fig. 4.

Value comparison. (A) Region of dmFC and adjacent ACC showing significant correlations with the V_unchosen − V_chosen value difference contrast. Additional areas correlating with this comparison signal are bilateral anterior insula and left dlFC. (B) Output of our stochastic decision model for the value comparison showing correlations with activity in the same brain regions. (C) The model explains activity in dmFC even on a subset of trials where subjects clearly choose the “correct” and not “erroneous” choice (where Vchosen − Vunchosen >0.2). This suggests that the result in (B) cannot be fully explained by error monitoring. (D) Average beta values in the random effects analysis of the model described in the text showing that neural activity in dmFC/ACC is explained better by the output of our decision model than by a decision difficulty based index of decision conflict (P < 10⁻⁷). The vertical lines, the SEM. (E) Illustration of the different stages involved in action based decision-making: action-based decision-making requires the computation of distinct value representations for both choice alternatives (purple box). These action values are compared against each other in a decision comparator (yellow box) to decide on a particular action. Such a comparator could yield a signal that approximately resembles the difference in the action values of the two actions. The output from this comparator could then be passed through a nonlinear function to inhibit a response of the unchosen action in primary motor areas (green box). The value of the chosen action is used to update future action values on the basis of experience, and to generate prediction errors (red box).