Defining and Differentiating Repeatability in Monte Carlo Simulation Studies
Year: 2026
Type: preprint
Abstract: Monte Carlo simulation studies are a primary methodology used to evaluate statistical methods which then are used in psychology and other social-behavioral research. While the replication crisis in social-behavioral science led to deep reflection on the repeatability of research studies and findings, the same is just beginning for Monte Carlo simulation studies. Recent evidence suggests there are reasons for concern about the repeatability of Monte Carlo simulations, which could have major implications for social-behavioral sciences more broadly, given the impact these studies have on analytical approaches used across fields. In this paper, we adopt a specific approach to describing and differentiating repeatability terms (replication, reproduction, and robustness; Nosek et al., 2025) and explore how these concepts should be applied and revised for the context of Monte Carlo simulations. We suggest key deviations from the Nosek et al. (2025) framework and a subdivision of repeatability types stemming from the potential for more systematic exploration of the role of sampling variability in simulation research through the sharing of open code. This approach demonstrates the importance and distinct value of each type of repeatability, as well as identifies the role of open data and code, and other verification studies, like multiverse analyses and registered reports. Monte Carlo simulation studies are important for informing statistical practice, and ensuring the repeatability of these studies is tantamount to the appropriate use of statistical methods in psychology and beyond.
Authors: Amanda Kay Montoya, Samantha F. Anderson
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7208834611