When Calibration Data Become Training Data: Specialization and Self-Generated Text in Learned Low-Rank Compression of Large Language Models
Year: 2026
Type: article
Abstract: Low-rank compression shrinks large language models by factorizing their weights, guided by a small calibration set. When the factorization is learned by distillation, as in learned subspace projections (LSP), the calibration set becomes training data, yet is still chosen by convention. We compressed OPT-1.3B, Llama-3.2-1B, and Qwen3-1.7B by 30% and 50% with LSP, calibrated on six sources at a matched budget (WikiText-2, C4, SlimPajama, Alpaca, and two kinds of self-generated text) over three seeds, and repeated the comparison on Llama-3.1-8B and with a second compressor. Training removed most of the up to 600-fold perplexity differences between untrained models but turned them into specialization: each source best preserved its own text, and WikiText-2, the customary choice, gave the least general models. Zero-shot accuracy slightly favored instruction data, through content rather than template. Self-generated documents gave the most uniformly faithful Llama models without external data, whereas exact samples of the model, whose expected training objective is the sequence-level divergence, failed on Qwen3-1.7B. The initialization’s domain survived training, and compressed Llama models collapsed without their calibration beginning-of-sequence token. With every source, compression lost most instruction following and multi-step reasoning. These results indicate that calibration sources should be reported with every perplexity.
Source: Informatics
Author: Van Tien Pham
Institution: Ton Duc Thang University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,172
OpenAlex ID: W7236574012
