Work

Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models

Year: 2025

Type: conference-paper

Abstract: Parameter-efficient fine-tuning (PEFT) has become a common method for fine-tuning large language models, where a base model can serve multiple users through PEFT module switching.To enhance user experience, base models require periodic updates.However, once updated, PEFT modules fine-tuned on previous versions often suffer substantial performance degradation on newer versions.Re-tuning these numerous modules to restore performance would incur significant computational costs.Through a comprehensive analysis of the changes that occur during base model updates, we uncover an interesting phenomenon: continual training primarily affects task-specific knowledge stored in Feed-Forward Networks (FFN), while having less impact on the task-specific pattern in the Attention mechanism.Based on these findings, we introduce Trans-PEFT, a novel approach that enhances the PEFT module by focusing on the task-specific pattern while reducing its dependence on certain knowledge in the base model.Further theoretical analysis supports our approach.Extensive experiments across 7 base models and 12 datasets demonstrate that Trans-PEFT trained modules can maintain performance on updated base models without re-tuning, significantly reducing maintenance overhead in real-world applications 1 .

Source: Annual Meeting of the Association for Computational Linguistics (ACL)

DOI: 10.18653/v1/2025.acl-long.719

Authors: Naibin Gu, Peng Fu, Xiyu Liu, Ke Ma, Zheng Lin, Weiping Wang

Institutions: Chinese Academy of Sciences, Institute of Information Engineering, University of Chinese Academy of Sciences

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: National Natural Science Foundation of China

Award: G2398966560