Extrapolating the Instruct Task Vector of Qwen2.5-0.5B: Scaling λ Beyond 1 Under Validation-Selected Coefficients on ARC, HellaSwag, and WinoGrande
Year: 2026
Type: preprint
Abstract: Task arithmetic adds a scaled task vector, the difference between a fine-tuned and a pre-trained model, to the pre-trained weights. We ask whether extrapolating past the fine-tuned model (λ>1) improves accuracy when λ is chosen on held-out validation data. We scale the instruct task vector of Qwen2.5-0.5B over seven values of λ up to 2 and measure macro accuracy on ARC-Easy, ARC-Challenge, HellaSwag and WinoGrande, with 1,199 validation items for selection and 3,967 disjoint test items. The pre-registered hypothesis, that some λ>1 beats λ=1, is not confirmed. The best extrapolated coefficient on validation (λ=1.5) is 0.78 percentage points (pp) below the instruct model on test, the primary contrast (95% bootstrap CI [-1.60,+0.06], p=0.070), and 0.38 pp below the base model, the secondary contrast (CI [-1.54,+0.82]). The setting λ=2 is clearly worse (-2.28 pp, CI [-3.35,-1.16]), and validation selects λ^*=0, the base model. The test peak at λ=0.75 is exploratory and indistinguishable from the instruct model. The results cover one 0.5B model pair on knowledge and reasoning benchmarks.Preprint. The attached archive contains the LaTeX source, the per-item evaluation caches, all analysis scripts and the model revisions needed to regenerate every number, table and figure (git tag preprints-v1; entry point run_everything.sh). Author: Dheiver Francisco Santos, ORCID 0000-0002-8599-9436. License: CC BY 4.0.
Source: Figshare
Author: Dheiver Francisco Santos
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7233600448
