Work

Exact Reproduction, an Untested Promotion Path, and a Judge That Never Answered: A Pre-registered Study of MLOps Evidence on a Local Stack

Year: 2026
Type: preprint
Abstract: Two pieces of evidence decide whether a machine learning team can be trusted near consequential decisions: a tracked, registered model line whose published metrics survive a from-scratch rerun, and a standardized evaluation whose failure cases are published rather than trimmed. This study pre-registered both tests on a local stack, in a protocol committed as the repository's root commit before any measured run existed. Half 1 re-implemented a published credit scorecard protocol (credit-risk-lab, pinned at commit fee580d0) under MLflow tracking, trained four candidates on the frozen split (30,000 accounts; 21,000 train, 9,000 test), and applied a frozen promotion gate. The champion rerun reproduced the published metrics exactly: AUC 0.7810 against a published 0.7810 (frozen tolerance 0.005, measured delta 0.0000), KS 0.4196 against 0.4196, Brier 0.1355 against 0.1355, and the logistic regression baseline also reproduced exactly (AUC 0.7518). H1 is CONFIRMED. The gate blocked both candidates it was given: the tuned challenger (AUC 0.7756) missed the frozen bar (champion + 0.003 = 0.7840), and the deliberately degraded negative control (labels permuted, AUC 0.4828) failed both frozen criteria. No candidate qualified, so the gate's promote direction was never exercised and H2 is INCONCLUSIVE under the frozen verdict logic; the challenger was not re-tuned to manufacture a pass. Half 2 built a complete Ragas evaluation of an extractive retrieval pipeline (rag-foundation, pinned at commit 007e3323) over a frozen 40-question set (28 in-corpus, 12 adversarial), and local generation already showed the targeted failure mode: the pipeline returned non-empty answers to 10 of the 12 adversarial questions whose correct behaviour is to decline. Scoring never ran: the single judge and embedding route the protocol permits returned HTTP 429 on every generateContent attempt, 21 of 21 logged calls after three successful wiring smoke calls, across the registered backoff ladder and spaced probes, and the run was stopped at 0 of 160 metric jobs. H3 and H4 are therefore NOT RUN, and no Ragas score is claimed anywhere in this paper. For this preprint, Half 1 was additionally re-executed from scratch in a clean MLflow store: every candidate's metrics and calibration table reproduced the committed record exactly, and the gate reproduced both block decisions with identical reasons.
Language: English
Open Access status: green
OpenAlex ID: W7236295302
Cites:
Cited by:
Related to: 10