Reliability in fake news studies: datasets and metrics
##plugins.themes.bootstrap3.article.main##
##plugins.themes.bootstrap3.article.sidebar##
Published
12-06-2024
Vianny Geraldine Castellanos-Trujillo
Patricia Palomares-Sánchez
Raúl Rodríguez-Ferrándiz
Tatiana Hidalgo Marí
https://orcid.org/0009-0009-7952-0116
Patricia Palomares-Sánchez
Raúl Rodríguez-Ferrándiz
Tatiana Hidalgo Marí
Abstract
Many empirical studies on disinformation use fake news datasets and real news datasets to compare them. Such studies often appeal to reliability (or confidence, trustworthiness, credibility, accuracy) as a factor to characterize and evaluate the message, medium, source and perception of the audience. We selected the 50 most cited articles in WoS and Scopus (2017-2022) on fake news and reliability and investigated where the researchers extracted their samples. That is, we propose a meta-research aimed at evaluating how reliable such samples are. We extract and analyze the criteria, indices, metrics, resources and databases that researchers declare to justify their selection of both authentic and false news.
How to Cite
Castellanos-Trujillo, V. G. ., Palomares-Sánchez, P., Rodríguez-Ferrándiz, R. ., & Hidalgo Marí, T. (2024). Reliability in fake news studies: datasets and metrics. ZER - Journal of Communication Studies, 29(56), 87–109. https://doi.org/10.1387/zer.26185
##plugins.themes.bootstrap3.article.details##
References
Allcott, H.; Gentzkow, M. (2017). Social media and fake news in the 2016 election. Journal of economic perspectives, 31(2), 211-236. https://doi.org/10.1257/jep.31.2.211
Alonso García, S., Gómez García, G., Sanz Prieto, M., Moreno Guerrero, A. J., & Rodríguez Jiménez, C. (2020). The impact of term fake news on the scientific community. Scientific performance and mapping in web of science. Social Sciences, 9(5), 73. https://doi.org/10.3390/socsci9050073
Appelman, A., & Sundar, S. S. (2016). Measuring Message Credibility: Construction and Validation of an Exclusive Scale. Journalism & Mass Communication Quarterly, 93(1), 59-79. https://doi.org/10.1177/1077699015606057
Barrera, O., et al. (2020). Facts, alternative facts, and fact checking in times of post-truth politics. Journal of Public Economics, 182, 104-123. https://doi.org/10.1016/j.jpubeco.2019.104123
Boukes, M., Jones, N. P., & Vliegenthart, R. (2022). Newsworthiness and story prominence: How the presence of news factors relates to upfront position and length of news stories. Journalism, 23(1), 98–116. https://doi.org/10.1177/1464884919899313
Casero-Ripollés, A. (2020). Impact of Covid-19 on the media system. Communicative and democratic consequences of news consumption during the outbreak. El profesional de la información, 29(2), e290223. https://doi.org/10.3145/epi.2020.mar.23
Clayton, K. Et. Al. (2020). Real Solutions for Fake News? Measuring the Effectiveness of General Warnings and Fact‑Check Tags in Reducing Belief in False Stories on Social Media. Political Behavior, 42, 1073–1095. https://doi.org/10.1007/s11109-019-09533-0
Del Vicario, M. Et Al. (2019). Polarization and Fake News: EarlyWarning of Potential Misinformation Targets. ACM Transactions on the Web, 13(2) https://doi.org/10.1145/3316809
Duffy, A., Tandoc, E., & Ling, R. (2020). Too good to be true, too good not to share: the social utility of fake news. Information, Communication & Society, 23(13),1965-1979. https://doi.org/10.1080/1369118X.2019.1623904
Edgerly, S. & Vraga, E. K. (2020). That’s Not News: Audience Perceptions of “News-ness” and Why It Matters. Mass Communication and Society, 23(5), 730–754. https://doi.org/10.1080/15205436.2020.1729383
Flanagin, A.J., Winter, S. & Metzger, M.J. (2020) Making sense of credibility in complex information environments: the role of message sidedness, information source, and thinking styles in credibility evaluation online, Information, Communication & Society, 23(7), 1038-1056. https://doi.org/10.1080/1369118X.2018.1547411
Floridi, L., & Illary, Ph. (2014). The philosophy of information quality. Springer.
Guess, A., Nyhan, B., & Reifler, J. (2018). Selective Exposure to Misinformation: Evidence from the consumption of fake news during the 2016 U.S. presidential campaign. European Research Council, 9.
Hovland, C. I., & Weiss, W. (1951). The influence of source credibility on communication effectiveness. Public Opinion Quarterly, 15, 635–650.
Herrero-Beaumont, E. (2022). Emerging Transparency Systems for News Governance to Protect Media Independence and Credibility in the Digital Infosphere. Communication Law and Policy, 27(3-4), 220-249. https://doi.org/10.1080/10811680.2022.2154071
Lawrence, R. G., Napoli, P. M. (eds.) (2023). News quality in the digital age. London: Routledge.
Lee, S., Gil De Zúñiga, H., & Munger, K. (2023). Antecedents and consequences of fake news exposure: a two-panel study on how news use and different indicators of fake news exposure affect media trust. Human Communication Research, hqad019. https://doi.org/10.1093/hcr/hqad019
Lee, Y. W., Strong, D.M., Kahn, B.K., & Wang, R.Y. (2002). AIMQ: a methodology for information quality assessment. Information & Management, 40, 133–146. https://doi.org/10.1016/S0378-7206(02)00043-5
Luo, M., Hancock, J. T., & Markowitz, D. M. (2022). Credibility Perceptions and Detection Accuracy of Fake News Headlines on Social Media: Effects of Truth-Bias and Endorsement Cues. Communication Research, 49(2), 171–195. https://doi.org/10.1177/0093650220921321
Lyons, B. A., Montgomery, J. M., Guess, A. M., Nyhan, B., & Reifler, J. (2021). Overconfidence in news judgments is associated with false news susceptibility. Proceedings of the National Academy of Sciences, 118(23), e2019527118.https://doi.org/10.1073/pnas.2019527118
Maertens, R., Roozenbeek, J., Basol, M., & Van Der Linden, S. (2021). Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments. Journal of Experimental Psychology: Applied, 27(1), 1–16. https://doi.org/10.1037/xap0000315
Mcquail, D. (1992). Media Performance: Mass Communication in the Public Interest. London: Sage.
Metzger, M. J. Et Al. (2003). Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association, 27(1), 293-335. https://doi.org/10.1080/23808985.2003.11679029
Metzger, M. J., & Flanagin, A. J. (2015). Psychological approaches to credibility assessment online. In S. S. Sundar (Ed.), The handbook of the psychology of communication technology (pp. 445–466). West Sussex, UK: John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118426456.ch20
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60(3), 413–439. https://doi.org/10.1111/j.1460-2466.2010.01488.x
Mitchell, A., Jurkowitz, M., Oliphant, J. B., & Shearer, E. (2020, July 30). Americans Who Mainly Get Their News on Social Media Are Less Engaged, Less Knowledgeable. Pew Research Center. https://pewrsr.ch/437CXY1
Nyhan, B. Et Al. (2020). Taking fact‑checks literally but not seriously? The effects of journalistic fact‑checking on Factual Beliefs and Candidate Favorability. Political Behavior 4, pp. 939–960. https://doi.org/10.1007/s11109-019-09528-x
Osmundsen, M. Et Al. (2021). Partisan Polarization Is the Primary Psychological Motivation behind Political Fake News Sharing on Twitter. American Political Science Review, 115(3), 999–1015. https://doi.org/10.1017/S0003055421000290
Palau-Sampio, D., Carratalá, A., Tarullo, R., & Crisóstomo, P. (2022). Quality recognition as a prescriber against disinformation. Comunicar, 72, 59-70. https://doi.org/10.3916/C72-2022-05
Park, A. Et Al. (2020). Understanding Fake News: A Bibliographic Perspective. Defence Strategic Communications, 8. https://doi.org/10.30966/2018.RIGA.8.4.
Pennycook, G., Epstein, Z. Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021). Shifting attention to accuracy can reduce misinformation online. Nature, 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2
Pennycook, G., Mcphetres, J., Zhang, Y., Lu, J. G., Rand, D. G. (2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7), 770–780. https://doi.org/10.1177/0956797620939054
Petersen M. B. Osmudsen, M., & Arceneaux, K. (2023). The “Need for Chaos” and Motivations to Share Hostile Political Rumors. American Political Science Review, 117(4), 1486–1505. https://doi.org/10.1017/S0003055422001447
Righetti, N. (2021). Four years of fake news: A quantitative analysis of the scientific literature. First Monday, 26(6-7). https://doi.org/10.5210/fm.v26i7.11645
Roozenbeek, J. & Van Der Linden, S. (2019). Fake news game confers psychological resistance against online misinformation. Palgrave Communication, 5, 65. https://doi.org/10.1057/s41599-019-0279-9
Roberts, C. (2010). Correlations among variables in message and messenger credibility scales. American Behavioral Scientist, 54(1), 43–56. https://doi.org/10.1177/0002764210376310
Sádaba, C. & Salaverría, R. (2023). Combatir la desinformación con alfabetización mediática: análisis de las tendencias en la Unión Europea. Revista Latina de Comunicación Social, 81, 17-33. https://www.doi.org/10.4185/RLCS-2023-1552
Serrano-Puche, J. (2017), Credibilidad y confianza en los medios de comunicación: un panorama de la situación en España. En González, M. y Valderrama, M. (coords.) (2017). Discursos comunicativos persuasivos hoy (pp. 427-438). Madrid: Tecnos.
Shearer, E. & Mitchell, A. (2021, January 12). News use across social media platforms in 2020. Pew Research Center. https://t.ly/S6tSd
Shin, I., Wang L., & Lu, Y. (2022). Twitter and Endorsed (Fake) News: The Influence of Endorsement by Strong Ties, Celebrities, and a User Majority on Credibility of Fake News During the COVID-19 Pandemic. International Journal of Communication, 16, 2573–2595.
Shu, K. Et Al. (2017). Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter 19, 24–36. https://doi.org/10.1145/3137597.3137600
Sterrett, D. Et Al. (2019). Who Shared It?: Deciding What News to Trust on Social Media. Digital Journalism, 7(6), 783-801. https://doi.org/10.1080/21670811.2019.1623702
Tandoc Et Al. (2018). Audiences’ acts of authentication in the age of fake news: A conceptual framework. New Media & Society, 20(8) 2745–2763.
Trillo-Domínguez, M., Salaverría, R., Codina, L., & De-Moya-Anegón, F. (2023). SCImago Media Rankings (SMR): situation and evolution of the digital reputation of the media worldwide. Profesional De La información, 32(5).
Trillo-Dominguez Et Al. (2024). Digital reputation indicator: A webometric approach for a global ranking of digital media, Journalism. https://doi.org/10.1177/14648849241237647
Truong, B.T., Allen, O.M. & Menczer, F. (2024). Account credibility inference based on news-sharing networks. EPJ Data Sci. 13(10). https://doi.org/10.1140/epjds/s13688-024-00450-9
Viviani, M., & Pasi, G. (2017). Credibility in social media: opinions, news, and health information—a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7. https://doi.org/10.1002/widm.1209
Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359, 1146–1151. https://www.science.org/doi/10.1126/science.aap9559
Zhou, X. & Zafarani, R. (2020). A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. ACM Computing Surveys, 53(5). https://doi.org/10.1145/3395046
Alonso García, S., Gómez García, G., Sanz Prieto, M., Moreno Guerrero, A. J., & Rodríguez Jiménez, C. (2020). The impact of term fake news on the scientific community. Scientific performance and mapping in web of science. Social Sciences, 9(5), 73. https://doi.org/10.3390/socsci9050073
Appelman, A., & Sundar, S. S. (2016). Measuring Message Credibility: Construction and Validation of an Exclusive Scale. Journalism & Mass Communication Quarterly, 93(1), 59-79. https://doi.org/10.1177/1077699015606057
Barrera, O., et al. (2020). Facts, alternative facts, and fact checking in times of post-truth politics. Journal of Public Economics, 182, 104-123. https://doi.org/10.1016/j.jpubeco.2019.104123
Boukes, M., Jones, N. P., & Vliegenthart, R. (2022). Newsworthiness and story prominence: How the presence of news factors relates to upfront position and length of news stories. Journalism, 23(1), 98–116. https://doi.org/10.1177/1464884919899313
Casero-Ripollés, A. (2020). Impact of Covid-19 on the media system. Communicative and democratic consequences of news consumption during the outbreak. El profesional de la información, 29(2), e290223. https://doi.org/10.3145/epi.2020.mar.23
Clayton, K. Et. Al. (2020). Real Solutions for Fake News? Measuring the Effectiveness of General Warnings and Fact‑Check Tags in Reducing Belief in False Stories on Social Media. Political Behavior, 42, 1073–1095. https://doi.org/10.1007/s11109-019-09533-0
Del Vicario, M. Et Al. (2019). Polarization and Fake News: EarlyWarning of Potential Misinformation Targets. ACM Transactions on the Web, 13(2) https://doi.org/10.1145/3316809
Duffy, A., Tandoc, E., & Ling, R. (2020). Too good to be true, too good not to share: the social utility of fake news. Information, Communication & Society, 23(13),1965-1979. https://doi.org/10.1080/1369118X.2019.1623904
Edgerly, S. & Vraga, E. K. (2020). That’s Not News: Audience Perceptions of “News-ness” and Why It Matters. Mass Communication and Society, 23(5), 730–754. https://doi.org/10.1080/15205436.2020.1729383
Flanagin, A.J., Winter, S. & Metzger, M.J. (2020) Making sense of credibility in complex information environments: the role of message sidedness, information source, and thinking styles in credibility evaluation online, Information, Communication & Society, 23(7), 1038-1056. https://doi.org/10.1080/1369118X.2018.1547411
Floridi, L., & Illary, Ph. (2014). The philosophy of information quality. Springer.
Guess, A., Nyhan, B., & Reifler, J. (2018). Selective Exposure to Misinformation: Evidence from the consumption of fake news during the 2016 U.S. presidential campaign. European Research Council, 9.
Hovland, C. I., & Weiss, W. (1951). The influence of source credibility on communication effectiveness. Public Opinion Quarterly, 15, 635–650.
Herrero-Beaumont, E. (2022). Emerging Transparency Systems for News Governance to Protect Media Independence and Credibility in the Digital Infosphere. Communication Law and Policy, 27(3-4), 220-249. https://doi.org/10.1080/10811680.2022.2154071
Lawrence, R. G., Napoli, P. M. (eds.) (2023). News quality in the digital age. London: Routledge.
Lee, S., Gil De Zúñiga, H., & Munger, K. (2023). Antecedents and consequences of fake news exposure: a two-panel study on how news use and different indicators of fake news exposure affect media trust. Human Communication Research, hqad019. https://doi.org/10.1093/hcr/hqad019
Lee, Y. W., Strong, D.M., Kahn, B.K., & Wang, R.Y. (2002). AIMQ: a methodology for information quality assessment. Information & Management, 40, 133–146. https://doi.org/10.1016/S0378-7206(02)00043-5
Luo, M., Hancock, J. T., & Markowitz, D. M. (2022). Credibility Perceptions and Detection Accuracy of Fake News Headlines on Social Media: Effects of Truth-Bias and Endorsement Cues. Communication Research, 49(2), 171–195. https://doi.org/10.1177/0093650220921321
Lyons, B. A., Montgomery, J. M., Guess, A. M., Nyhan, B., & Reifler, J. (2021). Overconfidence in news judgments is associated with false news susceptibility. Proceedings of the National Academy of Sciences, 118(23), e2019527118.https://doi.org/10.1073/pnas.2019527118
Maertens, R., Roozenbeek, J., Basol, M., & Van Der Linden, S. (2021). Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments. Journal of Experimental Psychology: Applied, 27(1), 1–16. https://doi.org/10.1037/xap0000315
Mcquail, D. (1992). Media Performance: Mass Communication in the Public Interest. London: Sage.
Metzger, M. J. Et Al. (2003). Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association, 27(1), 293-335. https://doi.org/10.1080/23808985.2003.11679029
Metzger, M. J., & Flanagin, A. J. (2015). Psychological approaches to credibility assessment online. In S. S. Sundar (Ed.), The handbook of the psychology of communication technology (pp. 445–466). West Sussex, UK: John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118426456.ch20
Metzger, M. J., Flanagin, A. J., & Medders, R. B. (2010). Social and heuristic approaches to credibility evaluation online. Journal of Communication, 60(3), 413–439. https://doi.org/10.1111/j.1460-2466.2010.01488.x
Mitchell, A., Jurkowitz, M., Oliphant, J. B., & Shearer, E. (2020, July 30). Americans Who Mainly Get Their News on Social Media Are Less Engaged, Less Knowledgeable. Pew Research Center. https://pewrsr.ch/437CXY1
Nyhan, B. Et Al. (2020). Taking fact‑checks literally but not seriously? The effects of journalistic fact‑checking on Factual Beliefs and Candidate Favorability. Political Behavior 4, pp. 939–960. https://doi.org/10.1007/s11109-019-09528-x
Osmundsen, M. Et Al. (2021). Partisan Polarization Is the Primary Psychological Motivation behind Political Fake News Sharing on Twitter. American Political Science Review, 115(3), 999–1015. https://doi.org/10.1017/S0003055421000290
Palau-Sampio, D., Carratalá, A., Tarullo, R., & Crisóstomo, P. (2022). Quality recognition as a prescriber against disinformation. Comunicar, 72, 59-70. https://doi.org/10.3916/C72-2022-05
Park, A. Et Al. (2020). Understanding Fake News: A Bibliographic Perspective. Defence Strategic Communications, 8. https://doi.org/10.30966/2018.RIGA.8.4.
Pennycook, G., Epstein, Z. Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021). Shifting attention to accuracy can reduce misinformation online. Nature, 592, 590–595. https://doi.org/10.1038/s41586-021-03344-2
Pennycook, G., Mcphetres, J., Zhang, Y., Lu, J. G., Rand, D. G. (2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7), 770–780. https://doi.org/10.1177/0956797620939054
Petersen M. B. Osmudsen, M., & Arceneaux, K. (2023). The “Need for Chaos” and Motivations to Share Hostile Political Rumors. American Political Science Review, 117(4), 1486–1505. https://doi.org/10.1017/S0003055422001447
Righetti, N. (2021). Four years of fake news: A quantitative analysis of the scientific literature. First Monday, 26(6-7). https://doi.org/10.5210/fm.v26i7.11645
Roozenbeek, J. & Van Der Linden, S. (2019). Fake news game confers psychological resistance against online misinformation. Palgrave Communication, 5, 65. https://doi.org/10.1057/s41599-019-0279-9
Roberts, C. (2010). Correlations among variables in message and messenger credibility scales. American Behavioral Scientist, 54(1), 43–56. https://doi.org/10.1177/0002764210376310
Sádaba, C. & Salaverría, R. (2023). Combatir la desinformación con alfabetización mediática: análisis de las tendencias en la Unión Europea. Revista Latina de Comunicación Social, 81, 17-33. https://www.doi.org/10.4185/RLCS-2023-1552
Serrano-Puche, J. (2017), Credibilidad y confianza en los medios de comunicación: un panorama de la situación en España. En González, M. y Valderrama, M. (coords.) (2017). Discursos comunicativos persuasivos hoy (pp. 427-438). Madrid: Tecnos.
Shearer, E. & Mitchell, A. (2021, January 12). News use across social media platforms in 2020. Pew Research Center. https://t.ly/S6tSd
Shin, I., Wang L., & Lu, Y. (2022). Twitter and Endorsed (Fake) News: The Influence of Endorsement by Strong Ties, Celebrities, and a User Majority on Credibility of Fake News During the COVID-19 Pandemic. International Journal of Communication, 16, 2573–2595.
Shu, K. Et Al. (2017). Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter 19, 24–36. https://doi.org/10.1145/3137597.3137600
Sterrett, D. Et Al. (2019). Who Shared It?: Deciding What News to Trust on Social Media. Digital Journalism, 7(6), 783-801. https://doi.org/10.1080/21670811.2019.1623702
Tandoc Et Al. (2018). Audiences’ acts of authentication in the age of fake news: A conceptual framework. New Media & Society, 20(8) 2745–2763.
Trillo-Domínguez, M., Salaverría, R., Codina, L., & De-Moya-Anegón, F. (2023). SCImago Media Rankings (SMR): situation and evolution of the digital reputation of the media worldwide. Profesional De La información, 32(5).
Trillo-Dominguez Et Al. (2024). Digital reputation indicator: A webometric approach for a global ranking of digital media, Journalism. https://doi.org/10.1177/14648849241237647
Truong, B.T., Allen, O.M. & Menczer, F. (2024). Account credibility inference based on news-sharing networks. EPJ Data Sci. 13(10). https://doi.org/10.1140/epjds/s13688-024-00450-9
Viviani, M., & Pasi, G. (2017). Credibility in social media: opinions, news, and health information—a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 7. https://doi.org/10.1002/widm.1209
Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359, 1146–1151. https://www.science.org/doi/10.1126/science.aap9559
Zhou, X. & Zafarani, R. (2020). A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities. ACM Computing Surveys, 53(5). https://doi.org/10.1145/3395046
Section
Dossier
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
(c) UPV/EHU Press