Year: 2025
Type: conference-paper
Abstract: Large Language Models have shown remarkable capabilities in the NLP domain.Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks.However, generally, full fine-tuning is a computationally expensive job.To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA).However, LoRA and its variants employ re-parametrized additive updates.In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations.We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies.We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.
Authors: Harsh Bihany, Shubham Patel, Ashutosh Modi
Institution: Indian Institute of Technology Kanpur
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888204