Get started
Work

The Harness Decided the Result: Three Evaluation Defects that Invert a QLoRA Fine-Tuning Study

Year: 2026
Type: preprint
Abstract: We fine-tuned Qwen3-1.7B with 4-bit QLoRA on 26,262 mathematics problems using a single consumer laptop GPU, and measured a gain of +24.94 percentage points on GSM8K and +27.80 on MATH-500. Both gains were large, both were highly significant under McNemar's test, and both were wrong. After correcting three defects in the evaluation harness, none of which touched the model, the data or the trained adapter, the same comparison on the same 1,819 test items yields -19.11 points on GSM8K and -13.20 on MATH-500. The adapter does not improve mathematical reasoning in this setting. It degrades it. The swing attributable to the harness alone is 44.05 points on GSM8K and 41.00 on MATH-500. The defects are a hybrid thinking mode left enabled for the baseline arm only, a generation budget below what that arm required, and a lenient answer-extraction fallback that scored the last number in a truncated reasoning trace as if it were a final answer. They combine asymmetrically: 99.1% of baseline GSM8K generations were truncated against 0.8% of fine-tuned ones, so the harness handicapped the control arm and left the treatment arm untouched. An ablation separates the two main defects and shows they carry very different weight on the two benchmarks, accounting for 44.05 and 1.13 points of the GSM8K baseline swing against 30.00 and 16.00 on MATH-500. Most of it costs no inference, and we validate the procedure against a run whose result it must reproduce and does, exactly, on both benchmarks. A fourth defect, a per-batch generated length recorded as though it were per-example, made the recorded truncation statistics unreliable. Significance testing offered no protection at any point: the original and corrected results are both significant, in opposite directions. We release the forensic record for all seven evaluation arms and a checklist that would have caught every defect.
DOI: 10.5281/zenodo.22234520
Language: English
Open Access status: green
OpenAlex ID: W7204877262
Cites:
Cited by:
Related to: