Work

Dynamic BB-OPLoRA: Rigid Core, Flexible Border for Subspace-Aware Low-Rank Adaptation

Year: 2026
Type: book
Abstract: Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models through compact additive updates, but these updates can still interfere with pretrained directions that support broad model capabilities, causing catastrophic forgetting.The recent Orthogonal-Projection LoRA (OPLoRA) method protects dominant pretrained directions, but its rigid preservation rule can limit adaptation and hinder optimization near the spectral boundary.To address this, we introduce Dynamic Budgeted-Border Orthogonal-Projection LoRA (Dynamic BB-OPLoRA), a subspace-aware LoRA that replaces a strict preservation rule with a rigid-core/flexible-border decomposition of the top singular subspace of each pretrained weight matrix.The rigid core preserves the most dominant pretrained directions, while the flexible border allows controlled taskspecific adaptation near the spectral boundary without destabilizing the core.The border update is governed by a stiffness-weighted budget that is dynamically adjusted using a gradientderived pressure signal.We evaluate the method on commonsense reasoning, mathematical reasoning, and Python code generation.Experiments show that Dynamic BB-OPLoRA improves the stability-plasticity trade-off over LoRA and OPLoRA, achieving stronger adaptation while maintaining resistance to cross-domain forgetting. Dynamic BB-OPLoRA
Language: English
Open Access status: green
OpenAlex ID: W7202279810
FWCI:
Cites: 36
Cited by:
Related to: 10