Work

Selection, Not Scale: A Pre-Registered Ladder Study of a 0.6B Arabic MCQ Generator

Year: 2026
Type: preprint
Abstract: Can a 0.6B model write Arabic reading-comprehension multiple-choice questions as well as akilx, a public 1.7B generator? We fixed the decision rules before any run, built the system as a ladder of five changes on Qwen3-0.6B, and spent $17.13 in cloud spend. On 460 ARCD passages the primary goal failed under the pre-registered protocol (final system minus akilx -3.5 points, 95% interval [-8.9, +2.0]); with options shuffled (paper 1's protocol) the difference is +0.0 [-5.7, +5.7]; the +5 primary criterion is not met under either. We stopped as the pre-registered failure rule required, and that decision stands. Two findings hold under both protocols. First, test-time selection does almost all of the work: best-of-5 with a verifier adds 23.5 points [+18.0, +28.9] (23.0 shuffled), while evidence-first prompting, constrained format and DPO are close to zero; the final 0.6B system is not significantly different from a tuned 1.7B model (-0.2 [-5.7, +5.0]; +1.7 shuffled). Second, our measurements misled us in five documented ways: an underpowered dev set reversed a real +12.8-point effect; wrong answer keys passed a human pass and, in one case, both judges; the good metric rewards questions that leak their answer; two leakage measures disagreed about DPO; and option order alone, a protocol detail lost between two studies of the same group, moved the headline comparison by 3.5 points and changed its outcome category.
Language: English
Open Access status: green
OpenAlex ID: W7236682945
Cites:
Cited by:
Related to: 10