ReproMut: A Controlled Mutation Benchmark for Comparing Reproducibility Auditors on Small Machine-Learning Experiments (Preprint, draft v0.1)
Year: 2026
Type: preprint
Abstract: Exploratory preprint, draft v0.2. The synthetic development run was not preregistered and is not a confirmatory study. New in v0.2: a separately pre-registered external validation (E1) on independently written code, and a cross-platform replication of the synthetic benchmark. Computational reproducibility failures in machine-learning experiments often stem from small, mundane causes: unfixed random seeds, silently drifting dependency defaults, hidden configuration overrides, stale cached artifacts and inconsistent preprocessing. We present ReproMut, an executable benchmark that injects predeclared reproducibility mutations into small, CPU-only, fully synthetic ML experiments with known ground truth: 28 baseline experiments, 12 result-changing mutation operators and 7 semantics-preserving controls (452 cases), each confirmed by an independent execution oracle. The seed-only check detected 29.4% of result-changing mutations and the static manifest 72.8%; the dynamic auditor detected all 272 with no false positives on 180 controls. The static manifest missed all 74 mutations hidden outside the declared configuration. All 452 case-level outcomes were identical on Linux aarch64 (Python 3.10, scikit-learn 1.7.2) and macOS arm64 (Python 3.12, scikit-learn 1.9.1). In the pre-registered external validation (E1) on 30 randomly sampled, independently written scikit-learn 1.7.2 example scripts (199 mutation cases, ground truth decided by execution), the seed-only check again missed every non-seed change (recall 71-75%), the static manifest raised false alarms on the 13% of plausible 'breaking' changes that did not alter results (false-positive rate 6.5-7.7%), and 2 of 32 runnable scripts were not reproducible run-to-run before any mutation. E1 could not exercise hidden-state failures, so that synthetic finding remains untested on real code. Because the dynamic auditor and the oracle both re-execute, high dynamic recall is partly expected by construction.
Author: Misree, Hamjeth
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7225585942