Year: 2025
Type: conference-paper
Abstract: Large language models (LLMs) have revolutionized artificial intelligence, but their performance on specific tasks is often limited by knowledge boundaries.While fine-tuning techniques like low-rank adaptation (LoRA) aim to address this, they can suffer from overfitting.We propose flexible low-rank adaptation (Flexora), a novel method that automatically selects the most critical layers for fine-tuning to optimize performance across diverse downstream tasks.Flexora formulates layer selection as a hyperparameter optimization problem, employs unrolled differentiation for efficient solving, and identifies the most impactful layers based on optimized hyperparameters.Extensive experiments across various pre-trained models and natural language tasks demonstrate that Flexora consistently outperforms existing baselines.We provide theoretical insights and comprehensive ablation studies to elucidate the effectiveness of Flexora.Therefore, Flexora offers a robust solution to enhance LoRA finetuning for LLMs, potentially advancing the field of adaptive language model optimization.
Authors: Chenxing Wei, Yao Shu, Ying Tiffany He, Fei Yu
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945313