Work

Tiny Budgets, Big Gains: Parameter Placement Strategy in Parameter Super-Efficient Fine-Tuning

Year: 2025

Type: conference-paper

Abstract: In this work, we propose FoRA-UA, a novel method that, using only 1-5% of the standard LoRA's parameters, achieves state-ofthe-art performance across a wide range of tasks.Specifically, we explore scenarios with extremely limited parameter budgets and derive two key insights: (1) fix-sized sparse frequency representations approximate small matrices more accurately; and (2) with a fixed number of trainable parameters, introducing a smaller intermediate representation to approximate larger matrices results in lower construction error.These findings form the foundation of our FoRA-UA method.By inserting a small intermediate parameter set, we achieve greater model compression without sacrificing performance.We evaluate FoRA-UA across diverse tasks, including natural language understanding (NLU), natural language generation (NLG), instruction tuning, and image classification, demonstrating strong generalisation and robustness under extreme compression. 1

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2025.emnlp-main.321

Authors: Jinman Zhao, Xueyan Zhang, Jiaru Li, Jingcheng Niu, Yulan Hu, Erxue Min, Gerald Penn

Institutions: University of Toronto, University of Waterloo, Northwestern University, Northwestern University, Technische Universität Darmstadt, Renmin University of China, Baidu (China)

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4416036959

FWCI: 0.953

Cites:

Cited by: 1

Related to: 10