Peer Review and the Technical Soundness of Research: Evidence from Automated Audits of Political Science Working Papers
Year: 2026
Type: preprint
Abstract: Peer review is meant to select high quality work and make it better. To measure one dimension of quality (“technical soundness”), we develop method-specific checklists and a “universal checklist” derived from the published literature by a transparent, rule-based procedure. We then have a language-model coder apply these checklists to 798 working papers posted from 2020 to 2022 on three preprint servers and to 406corresponding published versions. A one standard deviation increase in preprint soundness is associated with a 10.3 percentage point increase in publication probability and higher journal placement. For the published/unpublished margin, both the universal and the method-specific checklists predict publication outcomes to a similar degree. A “naive” review by the same model, with no checklist, predicts publication aboutas well overall and less well for top-three journals, but the checklists catch issues this misses and add further predictive power. Published versions score modestly higher than preprints: 67% score higher at publication, and the mean overall score rises by +0.19 standard deviations. We release the checklists and the auditing system as a tool for authors.
Source: SocArXiv (OSF Preprints)
Authors: Tak-Huen Chau, Andrew T. Little
Language: English
Subfield: Library and Information Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7236195247
