Work

Winner's Curse in Validation-Selected Model Merges: A Resampling Study on the Qwen2.5-0.5B Interpolation Grid (ARC, HellaSwag, WinoGrande)

Year: 2026
Type: preprint
Abstract: Merge coefficients are usually picked on a small validation set. The maximum over many candidates inflates the winner's validation score (the winner's curse); we did not find a measurement of this inflation for model merging. We measure it on 11 linear interpolations θ_t=(1-t)θ_base+t,θ_inst between Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct, scored in bfloat16 by the macro average of four benchmarks (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande; 300 validation and about 1000 test items per task). We select t on 1000 bootstrap resamples of the validation set and compare the validation gain of the selected t over t=0 with its test gain. The mean validation gain is +0.64 pp and the mean test gain is +0.17 pp, a ratio of 3.8. The mean difference, +0.48 pp, comes from simple resampling. The double bootstrap (test items resampled too) gives a 95% dispersion interval of [-0.65,+1.77] pp, and 67% of draws are positive. The pure selection bias is +0.68 pp. The data are consistent with a winner's curse; its magnitude is unresolved and conditional on bfloat16. One model pair, one grid.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.
Source: Figshare
Language: English
Open Access status: green
OpenAlex ID: W7233549455
Cites: 31
Cited by:
Related to: