Year: 2026
Type: conference-paper
Abstract: Pruning is essential for the efficient deployment of Large Language Models (LLMs); however, it causes severe performance degradation due to the structural distortion induced by sparsity.Existing recovery strategies, such as LoRA, predominantly employ global finetuning, often overlooking the mechanistic root of this degradation: the layer-wise accumulation and amplification of local errors.To address this limitation, we propose LaCo (Layerwise Compensation), a framework that reorients the recovery paradigm from global adaptation to hierarchical representation alignment.By sequentially optimizing each layer to reconstruct the model's hidden states, LaCo effectively intercepts the error propagation chain at its source.Extensive experiments demonstrate that LaCo surpasses parameter-efficient baselines in both perplexity reduction and zeroshot reasoning.Notably, it reduces recoverytime memory usage to approximately 1/7 of the baseline and requires only 2,048 unlabeled samples to match a LoRA model trained on 50k examples-achieving a ∼ 25× improvement in data efficiency.
Authors: Yingen Liu, Fan Wu, Panxuyan, Ruihui Li, Zhuo Tang, Kenli Li
Institution: Hunan University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G2744607400, G4431121220, G4480278818
OpenAlex ID: W7166847149