What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research
Year: 2026
Type: preprint
Abstract: Recent debates about psychology’s credibility have renewed attention to validity threats. We argue that pilot studies can function as methodological due diligence by improving early decisions about design, measurement, manipulation, sampling, and analysis prior to a main study. We examine four types of validity: construct, internal, external, and statistical-conclusion validity. Across these domains, we show how piloting can help researchers assess whether materials and procedures function as intended, identify potential validity threats, and provide key design parameters for planning and model choice. However, we also caution that piloting can harm validity when treated as a consequence-free space in which design or analytic choices are made without disclosing the pilot evidence behind them, or when small, noisy pilots are used to justify overconfident inferences. We therefore argue that when pilots are used to justify substantive design or analytic choices, they should be treated as part of the evidential record and reported transparently. Greater transparency would enable readers to evaluate how pilot evidence shaped the final design of a study and, consequently, better assess the validity of its findings.
Source: PsyArXiv (OSF Preprints)
Authors: Yashvin Seetahul, Mahmoud Medhat Elsherif, Caroline Zygar-Hoffmann, Lukas Wallrich, Priya Silverstein, Bjørn Sætrevik, Ilse L. Pit, Hannah Loenneker, Neele Henriette Heiser, Isaac J Handley-Miner, Christopher James Graham, Yu Yang Chou, Brett Buttliere, Agata Bochynska, Julia Beitner, Mary Beth Neff
Language: English
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7223059352