Auditing the evaluation pipeline of a machine learning benchmark: data, code and results for a case study on the Vesuvius Challenge
Year: 2026
Type: software
Abstract: Data, code and results accompanying the manuscript "Auditing the evaluation pipeline of a machine learning benchmark: a case study on the Vesuvius Challenge". The deposit contains the final leaderboard of the competition for 1,391 teams with both scores; 47 submissions by the author with pipeline provenance; scroll-disjoint fold definitions at scroll and volume level; an inventory of the audited notebooks with decision and rationale; official-metric scores for thirty volumes under three post-processing reductions; and 18 Python scripts. run_all.py is the single entry point and reproduces the quantitative results that rest on the redistributed data; the code-state census additionally requires the 30 inference notebooks, which are not redistributed. The remaining scripts cover the leaderboard analysis and its coupling diagnostics, exact group-coverage probabilities, a numerical check of the threshold-invariance result, a code-state census, figure generation, split-regime generators for nnU-Net, index reconciliation, leaderboard reconstruction from the platform exports, and the installation sequence for the competition scoring package. Two notebooks accompany the scripts: one reproduces the analysis with tables and figures inline, the other performs the official-metric measurement on Kaggle. Competition imagery and annotations are not redistributed. The fold files contain scroll membership and volume identifiers only, a derived tabulation shared under the non-commercial terms of the source data. The imaging data derive from the EduceLab-Scrolls dataset (Parsons et al. 2023, 10.48550/arXiv.2304.02084); acquisitions were made at the European Synchrotron Radiation Facility (beamline BM18) and Diamond Light Source (beamline I12). All 35 files are at the top level; there are no subdirectories.
Author: GOLGIYAZ, SEDAT
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7202391494