Predicting the replicability of social and behavioural science claims in COVID-19 preprints
Year: 2024
Type: article
Abstract: Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.
Source: Nature Human Behaviour
Authors: Alexandru Marcoci, David P. Wilkinson, Ans Vercammen, Bonnie C. Wintle, Anna Lou Abatayo, Ernest Baskin, Henk Berkman, Erin M. Buchanan, Sara Capitán, Tabaré Capitán, Ginny Chan, Kent Jason G. Cheng, Tom Coupé, Sarah Dryhurst, Jianhua Duan, John E. Edlund, Timothy M. Errington, Anna Fedor, Fiona Fidler, James G. Field, Nicholas Fox, Hannah Fraser, Alexandra L. J. Freeman, Anca Hanea, Felix Holzmeister, Sanghyun Hong, Raquel Huggins, Nick Huntington-Klein, Magnus Johannesson, Angela M. Jones, Hansika Kapoor, John Kerr, Melissa Kline Struhl, Marta Kołczyńska, Yang Liu, Zachary Loomas, Brianna Luis, Esteban Méndez, Olivia Miske, Fallon Mody, Carolin Nast, Brian A. Nosek, E. Simon Parsons, Thomas Pfeiffer, W. Robert Reed, Jon Roozenbeek, Alexa R. Schlyfestone, Claudia R. Schneider, Andrew Soh, Zhongchen Song, Anirudh Tagat, Melba Tutor, Andrew H. Tyner, Karolina Urbanska, Sander van der Linden
Institutions: University of Nottingham, University of Cambridge, The University of Melbourne, The University of Queensland, Curtin University, Wageningen University & Research, Saint Joseph's University, University of Auckland, Harrisburg University of Science and Technology, Swedish University of Agricultural Sciences, Pennsylvania State University, University of Canterbury, University College London, Statistics New Zealand, Rochester Institute of Technology, Center for Open Science, West Virginia University, Centre of Excellence for Biosecurity Risk Analysis, Universität Innsbruck, Seattle University, Stockholm School of Economics, Texas State University, University of Connecticut, University of Otago, Massachusetts Institute of Technology, Institute of Political Studies, Polish Academy of Sciences, University of California, Santa Cruz, University of Stavanger, University of Virginia, Massey University, University of Hawaiʻi at Mānoa
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $12,290
Awards: DARPA SCORE PROGRAM - TO ESTABLISH AN INFRASTRUCTURE TO SUPPORT RAPID, SCALABLE ASSESSMENT OF CLAIMS IN RESEARCH, BEST PRACTICES FOR CONDUCTING REPLICATIONS AND REPRODUCTIONS FOR VALIDATION OF CONFIDENCE IN CLAIMS, AND A METHODOLOGY FOR DRAMATICALLY ACCELERATING THE VALUE, CREDIBILITY ASSESSMENT, AND APPLICATION OF SCIENTIFIC EVIDENCE FOR DECISION-MAKING., THE PURPOSE OF THIS AGREEMENT IS TO FUND RESEARCH IN FURTHERANCE OF THE MISSION OF THE DEFENSE ADVANCED RESEARCH PROJECTS AGENCY DARPA DEFENSE SCIENCES OFFICE SYSTEMATIZING CONFIDENCE IN OPEN RESEARCH AND EVIDENCE SCORE PROGRAM. TERM: A THE TERM OF THE BASE PERIOD FOR THIS AGREEMENT COMMENCES ON DECEMBER 01, 2019 AND CONTINUES THROUGH DECEMBER 31, 2020. OPTION 1, IF EXERCISED, COMMENCES ON THE EFFECTIVE DATE OF EXERCISE AND CONTINUES FOR A PERIOD OF 18 MONTHS., G3263530556, G3350857844, G4086441874, G498543760, G5652728663, G6708036262, G7233641586, G8215844777, Psychosocial Determinants and Biological Pathways to Healthy Aging (Pathways)
OpenAlex ID: W4405645755