Functionally dissociable influences on learning rate in a dynamic environment

Abstract

Maintaining accurate beliefs in a changing environment requires dynamically adapting the rate at which one learns from new experiences. Beliefs should be stable in the face of noisy data but malleable in periods of change or uncertainty. Here we used computational modeling, psychophysics, and fMRI to show that adaptive learning is not a unitary phenomenon in the brain. Rather, it can be decomposed into three computationally and neuroanatomically distinct factors that were evident in human subjects performing a spatial-prediction task: (1) surprise-driven belief updating, related to BOLD activity in visual cortex; (2) uncertainty-driven belief updating, related to anterior prefrontal and parietal activity; and (3) reward-driven belief updating, a context-inappropriate behavioral tendency related to activity in ventral striatum. These distinct factors converged in a core system governing adaptive learning. This system, which included dorsomedial frontal cortex, responded to all three factors and predicted belief updating both across trials and across individuals.

Figure 1

Task overview and theoretical predictions. A: Screenshots of the experimental task. Participants positioned a bucket, trying to predict where a bag would drop from an occluded helicopter. B: An example sequence of trials. Data points mark the location at which successive bags fell (yellow = rewarding outcome, gray = neutral outcome). Heavy dashed line marks the true generative mean, which had periods of stability with occasional change points. Cyan line marks the predictions of an approximate Bayesian model. Inset equation presents the model’s belief-updating rule (B_t = belief, X_t = observed outcome, α_t = learning rate on trial t). Vertical dashed line marks the boundary between a high-noise condition (left) and low-noise condition (right), reflected in different levels of stochastic variance around the generative mean. C: Two theoretical influences on learning rate across trials. Change-point probability (CPP) is elevated when an unexpectedly large prediction error occurs. Relative uncertainty (RU) is elevated subsequently and slowly decays as a more precise estimate of the current mean is reached. Inset equation shows how CPP and RU jointly determine the adaptive learning rate.

Figure 2

Behavioral results. A: Example data from one participant, illustrating the fit obtained using only a fixed-learning-rate term. Each data point represents a trial. A fixed learning rate implies a linear relationship between prediction error (current outcome minus previous prediction) and update (next prediction minus previous prediction), regardless of noise (light gray = low noise; dark gray = high noise). B: Illustration of the fit obtained with a change-point probability term, using the same data as in panel A. Here the learning rate is adaptive (non-linear) and depends on noise. C: Coefficients from the full regression-based analysis of behavioral data (see inset equation), with coefficients estimated for each participant individually. The four plotted coefficients correspond to a fixed learning rate (β₁, panel A), change-point probability (β₂, panel B), relative uncertainty (β₃), and reward value (β₄). Estimates of β₄ are scaled by a factor of 5 for visibility. Black markers show the results of fitting the regression model to simulated data generated by the approximate Bayesian model (“Optimal”) or by a model with a fixed learning rate (“Fixed LR”).

Figure 3

Effects of individual learning-rate variables on BOLD, tested concurrently in the same GLM (see text and Table S1).

Figure 4

Brain regions selectively sensitive to CPP, RU, or reward. A: Regions showing significant effects (corrected p<0.05, whole-brain permutation test) in the same direction for contrasts of CPP versus 0, CPP versus RU, and CPP versus reward. Warm colors represent positive effects and cool colors represent negative effects. B: Regions showing significant effects in the same direction for contrasts of RU versus 0, RU versus CPP, and RU versus reward. C: Mean±SEM BOLD time courses relative to large change points (CPP>0.5), obtained from 33-voxel spheres centered at peak voxels in Panel A. Sensitivity to CPP entails a response that peaks soon after a change point and then decays rapidly (see Fig. 1C and S2). D: Equivalent time courses for peak locations in Panel B. See Movie S1 for further details of change-point-aligned time courses. E: Regions showing significant effects in the same direction for contrasts of reward versus 0, reward versus CPP, and reward versus RU.

Figure 5

Conjunction of BOLD effects for multiple influences on learning rate. A: Regions showing significant effects (corrected p<0.05, whole-brain permutation test) of all three learning-rate-related variables: CPP, RU, and reward value. B: Across-participant relationship between behavioral effects and BOLD effects in each conjunction region. Results are plotted for the effects of normative factors (the sum of CPP and RU parameters; left) and effects of reward value (right). Points represent across-participant Pearson correlations (with bootstrapped 95% CIs) between the behavioral parameter and the BOLD effect in each ROI.

Figure 6

BOLD response correlated with residual variability in prediction-updating behavior. A: Significant clusters for residual behavioral update (corrected p<0.05, whole-brain permutation test) were seen in both inferior and superior DMFC. B: The same effect was tested in common adaptive learning-rate regions (see Fig. 5). Points show the median residual update effect in each region with bootstrapped 95% CIs. C: Trial-by-trial BOLD amplitudes from the regions in Panel A were significant predictors of belief-updating behavior. BOLD amplitudes from either superior or inferior DMFC were added to a behavioral regression model that also contained all other hypothesized predictors of belief updating (see Fig. 2), and regression coefficients were estimated separately for each participant. The BOLD term tended to receive positive coefficients across participants, both for amplitudes extracted from the superior DMFC region (vertical axis) and from the inferior DMFC region (horizontal axis).