Year: 2026
Type: conference-paper
Abstract: Knowledge forgetting is a central challenge when adapting LLMs to new tasks.Prior studies indicate that pretrained knowledge is concentrated in the principal singular subspace of pretrained weight W 0 ; so recent Low-Rank Adaptation (LoRA) variants initialize LoRA in the minor subspace to steer early updates away from principal directions and mitigate forgetting.However, we observe that during fine-tuning, the update direction progressively shifts from the minor to the principal subspace, which is called as Singular-subspace Drift (SD), thereby allocating more energy to the directions that carry pretrained knowledge and leaving a persistent risk of forgetting.To address this issue, we propose Singular-subspace Drift Controlled LoRA (SDC-LoRA), which constrains the growth of update energy in the principal singular subspace of W 0 and thus mitigate SD.SDC-LoRA proposes Principal Subspace Energy-Controlled Learning, using Spectral Calibration factor γ sc to selectively downscale gradients along the principal singular subspace of W 0 while keeping minorsubspace updates unchanged.Across extensive experiments with LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Chat on MetaMathQA and CodeFeedback, SDC-LoRA mitigates forgetting on MMLU, TruthfulQA, and HellaSwag while matching or improving GSM8K and Hu-manEval, offering a practical route to adapt LLMs without sacrificing prior knowledge.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G7231209260
OpenAlex ID: W7166845108