Accuracy prompts are a replicable and generalizable approach for reducing the spread of misinformation

Abstract

Interventions that shift users attention toward the concept of accuracy represent a promising approach for reducing misinformation sharing online. We assess the replicability and generalizability of this accuracy prompt effect by meta-analyzing 20 experiments (with a total N = 26,863) completed by our group between 2017 and 2020. This internal meta-analysis includes all relevant studies regardless of outcome and uses identical analyses across all studies. Overall, accuracy prompts increased the quality of news that people share (sharing discernment) relative to control, primarily by reducing sharing intentions for false headlines by 10% relative to control in these studies. The magnitude of the effect did not significantly differ by content of headlines (politics compared with COVID-19 related news) and did not significantly decay over successive trials. The effect was not robustly moderated by gender, race, political ideology, education, or value explicitly placed on accuracy, but was significantly larger for older, more reflective, and more attentive participants. This internal meta-analysis demonstrates the replicability and generalizability of the accuracy prompt effect on sharing discernment.

Fig. 1. Accuracy prompts significantly increase sharing discernment.

Meta-analytic estimate (via random effects meta-analysis) of the effect of accuracy prompts on sharing discernment across the 20 experiments analyzed in this paper. The coefficient on the interaction between condition and headline veracity and 95% confidence interval are shown for each study, and the meta-analytic estimate is shown with the red dotted line and blue diamond (positive values indicate that the treatment increased sharing discernment). We find significant heterogeneity in effect size across studies, Cochran’s Q test, Q(19) = 88.53, p < 0.001, I² = 78.5% (k = 20 independent studies).

Fig. 2. Accuracy prompts significantly decrease sharing intentions for false news.

Meta-analytic estimate (via random effects meta-analysis) of the effect of accuracy prompts on sharing of false news across the 20 experiments analyzed in this paper. The coefficient on the condition dummy (which captures the effect of the treatment on sharing of false headlines) and 95% confidence interval are shown for each study, and the meta-analytic estimate is shown with the red dotted line and blue diamond. We find no significant heterogeneity in effect size across studies, Cochran’s Q test, Q(19) = 23.33, p = 0.223, I² = 18.5% (k = 20 independent studies).

Fig. 3. Accuracy prompts do not significantly affect sharing intentions for true news.

Meta-analytic estimate (via random effects meta-analysis) of the effect of accuracy prompts on sharing of true news across the 20 experiments analyzed in this paper. The coefficient on the condition dummy when analyzing true headlines and 95% confidence interval are shown for each study, and the meta-analytic estimate is shown with the red dotted line and blue diamond. We find no significant heterogeneity in effect size across studies, Cochran’s Q test, Q(19) = 22.42, p = 0.264, I² = 15.3% (k = 20 independent studies).

Fig. 4. The accuracy prompt effect on sharing of a given headline is strongly correlated with that headline’s perceived accuracy.

Meta-analytic estimate (via random effects meta-analysis) of the item-level correlation between the accuracy prompt effect on sharing and the headline’s out-of-sample perceived accuracy rating. The correlation coefficient and 95% confidence interval are shown for each study, and the meta-analytic estimate is shown with the red dotted line and blue diamond. We find no significant heterogeneity in effect size across studies, Cochran’s Q test, Q(14) = 18.99, p = 0.165 (k = 15 independent studies).

Fig. 5. Accuracy prompts reduce sharing to the extent that headlines are perceived as inaccurate.

For each of the 357 headlines in the 15 experiments where out-of-sample accuracy ratings were available, the accuracy prompt effect (sharing when treated minus sharing in control) is plotted against the headline’s perceived accuracy. False headlines are shown in green, true headlines in orange. Dot sizes are proportional to sample size. Best-fit line and 95% confidence interval are shown.

Fig. 6. Accuracy prompts increase sharing discernment across the ideological spectrum.

Shown is sharing discernment in the control (red triangles) versus treatment (blue circles) as a function of liberal versus conservative ideology. The model fits for discernment in control and treatment, based on meta-analytic estimates of model coefficients, are shown with solid lines. The meta-analytic estimate of discernment in control and treatment at each level of conservatism (rounded to the nearest 0.25) are shown with dots. More representative samples from Lucid and YouGov are shown in the left panel; convenience samples from Amazon Mechanical Turk are shown in the right panel.

Fig. 7. Flow diagram for study selection.

Demonstration of study selection for our internal meta-analysis.