Work

Split-Sample Strategies for Avoiding False Discoveries

Year: 2017
Type: report
Abstract: Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments.We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample.When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample.When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP.We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.
Authors: Michael Anderson, Jeremy Magruder
Language: English
Open Access status: green
OpenAlex ID: W2724805291
Cites: 35
Cited by: 88
Related to: 10