Fighting misinformation on social media using crowdsourced judgments of news source quality

Abstract

Reducing the spread of misinformation, especially on social media, is a major challenge. We investigate one potential approach: having social media platform algorithms preferentially display content from news sources that users rate as trustworthy. To do so, we ask whether crowdsourced trust ratings can effectively differentiate more versus less reliable sources. We ran two preregistered experiments (n = 1,010 from Mechanical Turk and n = 970 from Lucid) where individuals rated familiarity with, and trust in, 60 news sources from three categories: (i) mainstream media outlets, (ii) hyperpartisan websites, and (iii) websites that produce blatantly false content ("fake news"). Despite substantial partisan differences, we find that laypeople across the political spectrum rated mainstream sources as far more trustworthy than either hyperpartisan or fake news sources. Although this difference was larger for Democrats than Republicans-mostly due to distrust of mainstream sources by Republicans-every mainstream source (with one exception) was rated as more trustworthy than every hyperpartisan or fake news source across both studies when equally weighting ratings of Democrats and Republicans. Furthermore, politically balanced layperson ratings were strongly correlated (r = 0.90) with ratings provided by professional fact-checkers. We also found that, particularly among liberals, individuals higher in cognitive reflection were better able to discern between low- and high-quality sources. Finally, we found that excluding ratings from participants who were not familiar with a given news source dramatically reduced the effectiveness of the crowd. Our findings indicate that having algorithms up-rank content from trusted media outlets may be a promising approach for fighting the spread of misinformation on social media.

Keywords: fake news; media trust; misinformation; news media; social media.

Fig. 1.

Average trust ratings for each source among Democrats (x axis) and Republicans (y axis), in study 1 run on MTurk (A) and study 2 run on Lucid (B). Sources that are trusted equally by Democratic and Republican participants would fall along the solid line down the middle of the figure; sources trusted more by Democratic participants would fall below the line; and sources trusted more by Republican participants would fall above the line. Source names are shown for outlets with 33% familiarity or higher in study 1 and 25% familiarity or higher in study 2, when equally weighting Democratic and Republican participants. Dems, Democrats; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.

Fig. 2.

Average trust ratings given by professional fact-checkers (n = 8) for each source in study 2.

Fig. 3.

Average trust ratings for each source among professional fact-checkers and Democratic (A) or Republican (B) participants in study 2, run on Lucid. Dems, Democrats; FCers, professional fact-checkers; HuffPo, Huffington Post; Reps, Republicans; SFChronicle, San Francisco Chronicle; WashPo, Washington Post; WSJ, Wall Street Journal.