Distrust in name only: News readers and fake sources
By Bernhard Clemm (European University Institute)
The fake news problem has made politicians, companies, and academics wonder: How can we separate the Internet’s informational wheat from the chaff? And who can do it? In a novel study, I exposed news readers to content from sources with different degrees of trustworthiness. The findings suggest that we should be cautious about relying on the crowd to judge news sources.
Since citizens deal with a political world that is for the most part “out of reach, out of sight, out of mind”, as Walter Lippmann remarked a century ago (Lippmann 1922, p. 18), they cannot always know which facts are true. They must, at least to some extent, trust others to tell them the truth about politics. The question is whether citizens can be trusted to trust the right others.
In reaction to the recent surge of fake news, fact-checking seemed the obvious answer. But fact-checking as a means of combating the spread of misinformation has its limitations. Inventing factual claims is always faster and cheaper than investigating them, and the potential for stories to escape checking is large.
Expert assessment of sources has been an alternative response: For example, the startup NewsGuard employs “trained journalists and experienced editors” to assign news sources a “red” or “green” rating. But social media platforms — the primary habitat of false news — only reluctantly integrate such expert judgement into their ecosystem. Facebook’s fact-checking partnership with multiple third parties started off in 2016. But such cooperations can be awkward as the departure of fact-checking organization Snopes shows.
Facebook has also taken a wisdom-of-the-crowd approach, by asking users whether they are familiar with a source, and if so, whether they trust it. Facebook’s rationale is to improve the quality of news by prioritizing “[P]ublications that the community rates as trustworthy” The question is: Can Facebook trust “the community”? In a recent study with American participants, psychologists Gordon Pennycook of the University of Regina and David Rand of the Massachusetts Institute of Technology asked people — just like Facebook does — whether they recognized, and how much they trusted, a range of 60 US news sources (Pennycook and Rand 2019).
Reassuringly, participants across the political spectrum rated mainstream outlets as more trustworthy than “hyperpartisan” sources or websites producing blatantly false content. The researchers also invited a group of professional fact-checkers to rate sources the same way. The agreement between experts and laypersons was high. Up-ranking content judged as trustworthy by the crowd seems like a promising approach, the authors conclude.
However, a yet unpublished survey I ran with Paul Bauer from the University of Mannheim reveals some caveats. A representative sample of 1,980 Germans, recruited through the survey company Respondi, completed our online questionnaire. We asked respondents about their familiarity and trust with eleven German sources. Seven of these were the most widely read German outlets, and two were hyperpartisan sites (similar to the study by Pennycook and Rand 2019).
In addition, we added two sources that were in fact non-existent. We gave them neutral and familiar-sounding names such as “Nachrichten 360” (which translates to “News 360”). Think of the tale of the “Denver Guardian”: Playing on the names of existing outlets like “Denver Post” and “The Guardian”, its made-up story about a dead FBI agent investigating Hillary Clinton garnered half a million shares on Facebook. This suggests that fake news sources might succeed when exploiting an illusion of familiarity.
Figure 1a shows participants’ average trust ratings for different sources in our study. Similar to the results in Pennycook and Rand (2019), all mainstream outlets — with the exception of Bild, a tabloid and the most widely read German newspaper — fared better than the hyperpartisan sites.
The made-up sources, too, are judged less trustworthy than the mainstream sources. But the differences are smaller. For example, the gap between the most trustworthy source (“Tagesschau”) and the least trustworthy made-up source (“Nachrichten 360”) is only 1.2 points on a 5-point scale. Differences get even smaller when only looking at ratings of people who indicated they were familiar with the source (Figure 1b).
Figure 1: Average trust ratings across news sources
These findings call into question the crowd’s ability to assess the trustworthiness of news sources. Crowds might be able to detect obviously fishy or overtly partisan outlets. But they might perform worse at detecting fake sources with more neutral sounding names. Asking for their familiarity does not improve judgements. Although it is unclear how exactly, Facebook seems to incorporate this measure of familiarity into their trustworthiness score. The comparison between Figures 1a and 1b should make as sceptic.
In addition, we claim that the “direct” media trust question, as asked by Facebook and in our study, is not a good measurement of media trust in the first place. People questioned whether a source can be trusted might think: trusted to do what? Researchers have conceived of trust as an expectation about another agent regarding a specific behavior (Bacharach & Gambetta 2001, Bauer 2014). The standard media trust question does not specify such behavior, and it is unclear what the question measures.
One specific behavior that people should expect from a source they trust is that it reports the truth. In other words, a consequence of trust should be believing in the truth of what the source reports. We wanted to know how the source of information influences this belief — and how well the “direct” trust question corresponds to belief of a source’s messages.
We therefore exposed all participants in our study to the same factual news report about migration and asked them whether they believed it to be true. The article concerned numbers on family reunification of refugees in Germany. The story was true, but little read. Hence, participants most likely would not know whether it was true.
To understand the impact of the source on participants’ beliefs, we randomly presented the report as published by the original source (German public broadcaster “Tagesschau”) to one half of participants, and as published by a made-up source (“Nachrichten 360”) to the other half. We found that participants’ belief of the report was indeed higher when presented as an article from “Tagesschau”. But the difference was not large: Just over 0.6 points on a 7-point scale. This difference can be seen in the right-hand panel of Figure 2.
We also asked the same groups of participants directly how much they trusted the respective source, just reading its name. (This was the same question as reported in Figure 1). Here, the difference between the two groups is much larger, as the left-hand panel of Figure 2 shows.
In sum, the standard trust question exaggerates how much people differentiate between the existing and the made-up source. It does not reflect people’s accuracy judgements when receiving information.
Figure 2: Trust in existing vs. made-up source compared to belief in truth of news report by existing vs. made-up source
Of course, this result deserves careful interpretation. If the report was about a widely accepted fact, people might not differentiate between sources because they already accept the presented information to be factual. However, we carefully selected a report that had few shares on social media and was not widely reported across news outlets.
Results from a previous study before the German elections in 2017 provides supporting evidence for the tendency of people to disregard sources in judging news content. In this experiment, I exposed participants to news either attributed to a mainstream professional or a fringe source. For three out of four reports, there was no difference in belief in the truth of the news at all (Clemm von Hohenberg 2018).
Recent research demonstrates that measuring trustworthiness with a direct question might be a good proxy for source trustworthiness as perceived by experts and could therefore be used by social media platforms to up- or down-rank sources. However, this measure might fail to detect sources with a professional-sounding name. More importantly, even if people say they find a source untrustworthy they might not process information coming from such sources accordingly. Social media platforms should take this into account when developing tools to combat the spread and acceptance of misinformation.
About the Author
Bernhard Clemm is a PhD Researcher at the European University Institute, Florence. He previously studied philosophy, economics and political science in Germany, France and the UK and worked in the tech and media industries for a brief spell. His current research interests are in political psychology and public opinion, with a focus on the Internet and factual (mis)beliefs. Specifically, he is studying how people process factual claims and what factors shape accuracy judgements, as well as the role of media trust.
References
Bacharach, Michael, and Diego Gambetta (2001). “Trust in Signs“, in: Cook, Karen (ed.), Trust in Society. New York, Russell Sage Foundation.
Bauer, P. C. (2017). “Conceptualizing Trust and Trustworthiness”. Revised version of working paper published in: Political Concepts Working Paper Series, №61, http://dx.doi.org/10.2139/ssrn.2325989.
Clemm von Hohenber, Bernhard (2018). „The Ocean of Possible Truth. Drivers and Consequences of News Accuracy Judgements Online“, Working Paper, SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3281038.
Lippmann, Walter (1922/1997). Public Opinion. New York, Free Press Paperbacks.
Pennycook, Gordon, and David G. Rand (2019). “Fighting misinformation on social media using crowdsourced judgments of news source quality.” Proceedings of the National Academy of Sciences, 116 (7), https://doi.org/10.1073/pnas.1806781116.