Work

Small Effects, Big Noise: A Methodological Audit of Module Evaluation in Small-Scale Crack Instance Segmentation

Year: 2026
Type: preprint
Abstract: Crack instance segmentation papers built on YOLO-seg routinely claim gains by contrasting a baseline against the same model plus a proposed module, yet their statistical reliability is rarely examined. We audit a representative module, a skeleton-aware label assigner (STAL), across three epoch budgets, three seeds, three calibers, and eight metrics, and replicate on a second dataset and a second, structurally different module. Sweeping its tolerance proves the module is not inert; yet no gain is usable. The paired effect stays below 0.9 pt and flips sign with budget and caliber; the cross-seed correlation flips sign on identical runs; and the minimum detectable effect exceeds the observed one by 2.7–13.3×. All seven domain metrics are undecidable, one quantised to 11.1 pt by its 9 crossing pairs, and with three seeds exact Wilcoxon cannot return p < 0.25 at any effect size. Both replications reproduce the resolution ceiling, not the effect. From these results we derive an eight-rule minimum discernible protocol: effect-size floor, fixed caliber, verified pairing, MDE beside every difference, per-seed dispersion, fixed budgets, computed seed budget; every rule checkable from a single per-seed table.
Author: Xin Cao
Language: English
Open Access status: green
OpenAlex ID: W7233911015
Cites: 8
Cited by:
Related to: 10