Year: 2025
Type: conference-paper
Abstract: As the scale of large language models (LLMs) grows and natural language tasks become increasingly diverse, Parameter-Efficient Fine-Tuning (PEFT) has become the standard paradigm for fine-tuning LLMs.Among PEFT methods, LoRA is widely adopted for not introducing additional inference overhead.However, existing LoRA's shared parameter space paradigm introduces parameter interference, leading to a gap in generalization performance for specific tasks compared to full fine-tuning.To address this issue, we propose a parameterseparated low-rank adapter, called Subspace Low-Rank Adaptation (SuLoRA).The core idea of SuLoRA is to account for task differences by decomposing LoRA's parameter matrix into multiple independent subspaces and assigning them differentially to distinct tasks.This prevents interference across tasks and enhances the effectiveness of low-rank adaptation.Additionally, SuLoRA achieves higher rank expansion by freezing the A matrix, further improving generalization capability.We conduct extensive experiments on various NLP tasks, demonstrating that SuLoRA significantly outperforms LoRA in trainable parameter efficiency and overall model performance.Furthermore, we validate SuLoRA's effectiveness in domain generalization and multi-modal tasks, showcasing its strong generalization ability.
Authors: Chenhao Ding, Jiangyang Li, SongLin Dong, Xinyuan Gao, Yuhang He, Yihong Gong
Institution: Xi'an Jiaotong University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G3038487828, G6408364431, G6441506416
OpenAlex ID: W4412888646