Conformist social learning leads to self-organised prevention against adverse bias in risky decision making

Abstract

Given the ubiquity of potentially adverse behavioural bias owing to myopic trial-and-error learning, it seems paradoxical that improvements in decision-making performance through conformist social learning, a process widely considered to be bias amplification, still prevail in animal collective behaviour. Here we show, through model analyses and large-scale interactive behavioural experiments with 585 human subjects, that conformist influence can indeed promote favourable risk taking in repeated experience-based decision making, even though many individuals are systematically biased towards adverse risk aversion. Although strong positive feedback conferred by copying the majority's behaviour could result in unfavourable informational cascades, our differential equation model of collective behavioural dynamics identified a key role for increasing exploration by negative feedback arising when a weak minority influence undermines the inherent behavioural bias. This 'collective behavioural rescue', emerging through coordination of positive and negative feedback, highlights a benefit of collective learning in a broader range of environmental conditions than previously assumed and resolves the ostensible paradox of adaptive collective behavioural flexibility under conformist influences.

Keywords: collective behaviour; computational biology; conformity; hot stove effect; human; physics of living systems; reinforcement learning; risky decision making; social learning; systems biology.

Figure 1.. Mitigation of suboptimal risk aversion by social influence.

(a) A schematic diagram of the task. A safe option provides a constant reward ${\pi }_s=1$ whereas a risky option provides a reward randomly drawn from a Gaussian distribution with mean $\mu =1.5$ and $\text{s.d.}=1$. (b, c): The emergence of suboptimal risk aversion (the hot stove effect) depending on a combination of the reinforcement learning parameters; (b): under no social influence (i.e. the copying weight $\sigma =0$), and (c): under social influences with different values of the conformity exponents $\theta$ and copying weights $\sigma$. The dashed curve is the asymptotic equilibrium at which asocial learners are expected to end up choosing the two alternatives with equal likelihood (i.e. $P_{r,t\to \mathrm{\infty }}=0.5$), which is given analytically by $\beta =(2-\alpha )/\alpha$(Denrell, 2007). The coloured background is a result of the agent-based simulation with total trials $T=150$ and group size $N=10$, showing the average proportion of choosing the risky option in the second half of the learning trials ${\displaystyle P_{r,t>75}>0.5}$ under a given combination of the parameters. (d): The differences between the mean proportion of risk aversion of asocial learners and that of social learners, highlighting regions in which performance is improved (orange) or undermined (purple) by social learning.

Figure 1—figure supplement 1.. The simulation result with a wider parameter space.

The effect of the relationship between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across the different combinations of social learning parameters on the mean proportion of choosing the risky alternative in the second half of the trials of the two-armed bandit task described in Figure 1 in the main text. The dashed curves give a set of parameter combinations with which asocial learners are expected to choose the risky alternative in the same proportion as they choose the safe alternative (i.e. $P_r^{\star }=0.5$) in the infinite time horizon $T\to \mathrm{\infty }$, given by $\beta =(2-\alpha )/\alpha$.

Figure 1—figure supplement 2.. The results of the value-shaping social influence model.

The relationships between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across different combinations of social learning parameters. The coloured background shows the average proportion of choosing the risky option in the second half of the learning trials ${\displaystyle P_{r,t>75}>0.5}$. Different social learning weights (${\sigma }_{vs}$) are shown from top to bottom (${\sigma }_{vs}\in \{0,0.1,0.25,0.5,1,2\}$). Different conformity exponents are shown from left to right ($\theta \in \{0.5,1,2\}$). The dashed curve is the asymptotic equilibrium at which asocial learners are expected to end up choosing both alternatives with equal likelihood (i.e. $P_r^{\star }=0.5$), given by $\beta =(2-\alpha )/\alpha$.

Figure 1—figure supplement 3.. The simulation result with the negative risk premium.

The relationships between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across different combinations of social learning parameters. The coloured background shows the average proportion of choosing the risky option in the second half of the learning trials ${\displaystyle P_{r,t>75}>0.5}$. Different social learning weights ($\sigma$) are shown from top to bottom ($\sigma \in \{0,0.25,0.5,0.75,0.9\}$). Different conformity exponents are shown from left to right ($\theta \in \{1,2,4,8\}$). The risk premium is negative$\mu =-0.5.$

Figure 1—figure supplement 4.. The simulation result with the Bernoulli noise distribution.

The relationships between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across different combinations of social learning parameters. The coloured background shows the average proportion of choosing the risky option in the second half of the learning trials ${\displaystyle P_{r,t>75}>0.5}$. Different social learning weights ($\sigma$) are shown from top to bottom ($\sigma \in \{0,0.2,0.4,0.6,0.8\}$). Different conformity exponents are shown from left to right ($\theta \in \{1,2,4,8\}$). The binary payoff distribution was used where the safe alternative always provides ${\pi }_s=1$ while the risky alternative provides either a 70% chance of ${\pi }_r=0$ or a 30% chance of ${\pi }_r=5$ . The risk premium was 1.5.

Figure 1—figure supplement 5.. The simulation results under the positive risk premium experimental setups (a,d: the 1-risky-1-safe; b,e: the 1-risky-3-safe; c,f: the 2-risky-2-safe).

The relationships between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across different combinations of social learning parameters. (a–c): The coloured background shows the average proportion of choosing the risky option in the second half of the learning trials (${\displaystyle P_{r,t>75}>0.5}$) under social influences with different values of the conformity exponents $\theta$ and copying weights $\sigma$. The dashed curve is the asymptotic equilibrium at which asocial learners are expected to end up choosing the two alternatives with equal likelihood (i.e. $P_r=0.5$). (d–f): The differences between the mean proportion of risk aversion of asocial learners and that of social learners, highlighting regions in which performance is improved (that is, risk-seeking increases; orange) or undermined (that is, risk-aversion is amplified; purple) by social learning.

Figure 1—figure supplement 6.. The simulation results under the negative risk premium experimental setup.

The relationships between individual learning rate ($\alpha$) and individual inverse temperature ($\beta$) across different combinations of social learning parameters. (left): The coloured background shows the average proportion of choosing the (optimal) safe option in the second half of the learning trials under social influences with different values of the conformity exponents $\theta$ and copying weights $\sigma$. The dashed curve shows the proportion of choosing the safe option at $P_s=0.85$. (right): The differences between the mean proportion of risk aversion of asocial learners and that of social learners, highlighting regions in which (suboptimal) risk-seeking increases (orange) and (optimal) risk-aversion increases (purple) by social learning.

Figure 2.. The effect of social learning on average decision performance.

The x axis is a product of two reinforcement learning parameters $\alpha (\beta +1)$, namely, the susceptibility to the hot stove effect. The y axis is the mean probability of choosing the optimal risky alternative in the last 75 trials in a two-armed bandit task whose setup was the same as in Figure 1. The black solid curve is the analytical prediction of the asymptotic performance of individual reinforcement learning with infinite time horizon $T\to +\mathrm{\infty }$ (Denrell, 2007). The analytical curve shows a choice shift emerging at $\alpha (\beta +1)=2$; that is, individual learners ultimately prefer the safe to the risky option in the current setup of the task when ${\displaystyle \alpha (\beta +1)>2}$. The dotted curves are mean results of agent-based simulations of social learners with two different mean values of the copying weight $\sigma \in \{0.25,0.5\}$ (green and yellow, respectively) and asocial learners with $\sigma =0$ (purple). The difference between the agent-based simulation with $\sigma =0$ and the analytical result was due to the finite number of decision trials in the simulation, and hence, the longer the horizon, the closer they become (Figure 2—figure supplement 1). Each panel shows a different combination of the inverse temperature $\beta$ and the conformity exponent $\theta$.

Figure 2—figure supplement 1.. The effect of social learning on the average decision performance on the longer time horizon.

The x axis is an interaction of two reinforcement learning parameters $\alpha (\beta +1)$, that is, the susceptibility to the hot stove effect. The y axis is the mean probability of choosing the optimal risky alternative in the last 75 trials in the two-armed bandit task whose setup was the same as in Figures 1 and 2 in the main text (i.e. $\mu =0.5$, s.d. = 1) except for the longer time horizon $T=1075$ compared to the time horizon used in the main text ($T=150$). The dotted curves are the mean result of agent-based simulations of groups of social learners with two different mean values of the copying weight $\sigma \in \{0.25,0.5\}$ or individual learners with $\sigma =0$. Each panel shows a different combination of the inverse temperature $\beta$ and the conformity exponent $\theta$. The black solid curve is the theoretical benchmark where individual reinforcement learners were expected to asymptote with $T\to +\mathrm{\infty }$. Compared to Figure 2 in the main text, individual learners got closer to the benchmark. On the other hand, the performance of social learners remained deviated from the benchmark, suggesting that social influence had a qualitative impact on the course of learning and decision making, rather than merely slowing down approaching the equilibrium of individual learning.

Figure 2—figure supplement 2.. The effect of social learning on the time evolution of decision performance.

The x axis is the number of trials. The y axis is the mean proportion of choosing the optimal risky alternative. Each colour shows a different $\beta$. For the asocial learning condition (i.e. $\sigma =0$), the analytical benchmark to which reinforcement learners asymptote is shown as a horizontal line. Conformity exponent $\theta$ was 2. Group size was 8. The simulation was repeated 1000 times for each combination of parameters. Compared to asocial learning cases, social learning ($\sigma =0.3$) qualitatively alters the course of learning, rather than just speeding up or slowing down learning.

Figure 3.. The effect of individual heterogeneity on the proportion of choosing the risky option in the two-armed bandit task.

(a) The effect of heterogeneity of $\alpha$, (b) $\beta$, (c) $\sigma$, and (d) $\theta$. Individual values of a focal behavioural parameter were varied across individuals in a group of five. Other non-focal parameters were identical across individuals within a group. The basic parameter values assigned to non-focal parameters were $\alpha =0.5$, $\beta =7$, $\sigma =0.3$, and $\theta =2$, and groups’ mean values of the various focal parameters were matched to these basic values. We simulated 3 different heterogeneous compositions: The majority (3 of 5 individuals) potentially suffered the hot stove effect ${\displaystyle {\alpha }_i({\beta }_i+1)>2}$ (a, b) or had the highest diversity in social learning parameters (c, d; purple); the majority were able to overcome the hot stove effect ${\displaystyle {\alpha }_i({\beta }_i+1)<2}$ (a, b) or had moderate heterogeneity in the social learning parameters (c, d; blue); and all individuals had ${\displaystyle {\alpha }_i({\beta }_i+1)>2}$ but smaller heterogeneity (green). The yellow diamond shows the homogeneous groups’ performance. Lines are drawn through average results across the same compositional groups. Each round dot represents a group member’s mean performance. The diamonds are the average performance of each group for each composition category. For comparison, asocial learners’ performance, with which the performance of social learners can be evaluated, is shown in gray. For heterogeneous $\alpha$ and $\beta$, the analytical solution of asocial learning performance is shown as a solid-line curve. We ran 20,000 replications for each group composition.

Figure 4.. The population dynamics model.

(a) A schematic diagram of the dynamics. Solid arrows represent a change in population density between connected states at a time step. The thicker the arrow, the larger the per-capita rate of behavioural change. (b, c) The results of the asocial, baseline model where $P_S^{-}=P_R^{+}=p_h$ and $P_R^{-}=P_S^{+}=p_l$ (${\displaystyle p_h>p_l}$). Both figures show the equilibrium bias towards risk seeking (i.e., $N_r^{\star }-N_s^{\star }$) as a function of the degree of risk premium $e$ as well as of the per-capita probability of moving to the less preferred behavioural option $p_l$. (b) The explicit form of the curve is given by $-n(p_h-p_l)\left\{(1-e)p_h-e{p}_l\right\}/(p_h+p_l)\left\{(1-e)p_h+e{p}_l\right\}$. (c) The dashed curve is the analytically derived neutral equilibrium of the asocial system that results in $N_R^{*}=N_S^{*}$, given by $e=p_h/(p_h+p_l)$. (d) The equilibrium of the collective behavioural dynamics with social influences. The numerical results were obtained with $N_{S,t=0}^{-}=N_{S,t=0}^{+}=5$, $N_{R,t=0}=10$, and $p_h=0.7$.

Figure 4—figure supplement 1.. The result of the differential equation model.

The effect of both the per capita probability of exploration $p_l$ and $e$ (i.e. the ratio of individuals who prefer behavioural state $R$) on the equilibrium degree of risk seeking (i.e. $N_R^{*}-N_S^{*}$), across the different combinations of social influence parameters. Different social influence weights are shown from top to bottom ($\sigma \in \{0,0.25,0.5,0.75\}$). Different conformity exponents are shown from left to right ($\theta \in \{1,2,10\}$). The dashed curve is $e=p_h/(p_h+p_l)$. The numeric solution was obtained with conditions $N_{S,t=0}^{-}=N_{S,t=0}^{+}=5$, $N_{R,t=0}=10$, and $p_h=0.7$.

Figure 5.. The approximate bifurcation analysis.

The relationships between the social influence weight $\sigma$ and the equilibrium number of individuals in the risky behavioural state $N_R^{\star }$ across different conformity exponents $\theta \in \{0,1,2,10\}$ and different values of risk premium $e\in \{0.55,0.65,0.7,0.75\}$, are shown as black dots. The background colours indicate regions where the system approaches either risk aversion (${\displaystyle N_R^{\star }<N_S^{\star }}$; blue) or risk seeking (${\displaystyle N_R^{\star }>N_S^{\star }}$; red). The horizontal dashed line is $N_R=N_S=10$. Two locally stable equilibria emerge when $\theta \ge 2$, which suggests that the system has a bifurcation when $\sigma$ is sufficiently large. The other parameters are set to $p_h=0.7$, $p_l=0.2$, and $N=20$.

Figure 5—figure supplement 1.. The approximate bifurcation analysis.

The relationship between the social influence weight $\sigma$ and the equilibrium number of individuals choosing the risky alternative $N_R^{\star }$ across the different conformity exponents $\theta (\in \{0,1,2,10\})$, shown as black dots. The triangular points shown in the background of each panel indicate regions in which the group approaches risk aversion (i.e., ${\displaystyle N_R^{\star }<10}$; blue) or the risk-seeking equilibrium (i.e. ${\displaystyle N_R^{\star }>10}$; red). Two different equilibria mean that the system has a bifurcation under a given $\sigma$. The direction of the background triangles indicates whether $N_R$ increases (${\displaystyle \mathrm{\Delta }}$) or decreases (${\displaystyle \mathrm{\nabla }}$) relative to its starting position. The other parameters are set to $p_h=0.7$, $p_l=0.2$.

Figure 6.. Prediction of the fit learning model.

Results of a series of agent-based simulations with individual parameters that were drawn randomly from the best fit global parameters. Independent simulations were conducted 100,000 times for each condition. Group size was fixed to six for the group condition. Lines are means (black-dashed: individual, coloured-solid: group) and the shaded areas are 80% Bayesian credible intervals. Mean performances of agents with different ${\sigma }_i$ are shown in the colour gradient. (a) A two-armed bandit task. (b) A 1-risky-3-safe (four-armed) bandit task. (c) A 2-risky-2-safe (four-armed) bandit task. (d) A negative risk premium two-armed bandit task.

Figure 6—figure supplement 1.. Experimental results with the mixed logit model regression.

The black triangles are subjects in the individual learning condition; the orange dots are those in the group condition with group sizes ranging from 2 to 8. The solid lines are predictions from a mixed logit model for the individual condition (black) and for the group condition (orange), with the shaded area showing the 95% Bayesian credible intervals (CIs). (a) A two-armed bandit task ($N=168)$. (b) A 1-risky-3-safe (four-armed) bandit task ($N=148$). (c) A 2-risky-2-safe (four-armed) bandit task ($N=151$). (d) A negative risk premium (RP) two-armed bandit task ($N=118$). The width of the CI for the individual condition in the negative RP task is due to the lack of data points in the region. The x axis is ${\alpha }_i({\beta }_i+1)$, namely, the susceptibility to the hot stove effect. (a, b, and d) The y axis is the mean proportion of choosing the risky alternative averaged over the second half of the trials. (c) The y axis is the mean proportion of choosing the optimal risky alternative averaged over the second half of the trials. The horizontal lines show the chance-level probability.

Figure 6—figure supplement 2.. Bayesian model comparison.

(a) The model recovery performance: model frequencies (dark shade) and exceedance probability (XP) for each pair of simulated and fitted models, calculated by the Widely Applicable Information Criterion (WAIC). (b–d) Model comparison results. The lengths of the bars indicate model frequencies. Exceedance probability (XP) of the decision-biasing model is shown.

Figure 6—figure supplement 3.. The parameter recovery performance.

The top half and bottom half of the figure are the results of parameter recovery test 1 and 2, respectively. The left column shows the global parameters fitted for each of the two four-armed bandit tasks, the 1-risky-3-safe task ($N=105$) and the 2-risky-2-safe task ($N=105$). The red points are the true values and the black points are the mean posterior values (i.e. recovered values). The 95% Bayesian credible intervals are shown with error bars. The middle and right column are individual-level parameters across the two task conditions ($N=210$). The x axis is the true value and the y axis is the fitted (i.e. the mean posterior) individual value. The differences between the true value and the estimated value are shown in different colours (Dark: fit well). The Pearson’s correlation coefficients between the true and fitted values are shown.