Evaluating Orthogonal Subspace and Sparse Low-Rank Adaptation Under Model-Directed Autonomous Curricula
Year: 2026
Type: preprint
Abstract: Orthogonality- and sparsity-based low-rank adaptation methods such as O-LoRA and LoRI are intended to reduce catastrophic forgetting in language models, yet, to our knowledge, they have only been evaluated on task sequences chosen by a person. Self-editing systems such as SEAL break that assumption: the model writes its own training data and can also decide what to learn next. We build a small, fully logged testbed to ask whether the protection these methods give survives when the model picks the order. Qwen2.5-1.5B-Instruct is edited sequentially on 19 facts from 2024 (38 held-out questions) under five configurations and three seeds, 15 runs and 285 edits in total. Vanilla LoRA finishes at 2.6% accuracy, below the untrained model's 7.9%, and damages facts the base model already partly knew more than any other method (50% to 5.6% on those facts), whereas LoRI-style training leaves them intact and O-LoRA-style training falls in between. The OLoRA-style and LoRI-style variants finish at 12.3–14.0%, about ten points above vanilla; the bootstrap intervals for that gap narrowly exclude zero, though we did not correct for multiple comparisons, so we call the evidence suggestive. The Curriculum Transfer Gap is +0.9 points for O-LoRA and −0.9 points for LoRI, roughly a third of one question, with intervals of about ±8 points, so neither can be told apart from zero. Model-chosen orders were no more topic-clustered than the human order and were biased toward the first-listed candidate. We present this as an exploratory, small-scale result, document where our implementations depart from the original methods, and describe a second protocol with control curricula that is implemented but not yet run.
Authors: Shaikh, Altamash, Gupta, Shiwani, Maha, Anand
Institution: Thakur College of Engineering and Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236711612
Cites:
Cited by:
Related to:
