Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning
Year: 2026
Type: conference-paper
Abstract: Parameter-efficient fine-tuning methods such as LoRA reduce trainable parameters, but still apply dense low-rank updates per token, leaving adaptation compute largely fixed once rank is set.We propose Context-Conditioned Masked LoRA (CCM-LoRA), which learns a lightweight router that activates an inputdependent subset of LoRA rank directions, turning LoRA into dynamic rank routing and enabling contextual sparsity in fine-tuning and inference.CCM-LoRA is trained with a budget-constrained objective that targets an expected effective rank (or FLOPs) while regularizing routing to avoid degenerate alwayson/off masks.Across public NLU and multilingual benchmarks, CCM-LoRA improves the accuracy-efficiency Pareto frontier versus static-rank LoRA and adaptive-rank baselines, matching or improving task performance at lower inference-time effective rank.We also provide a reproducible profiling protocol and analyses of rank usage, router overhead, and robustness under domain and language shift.
Authors: Rifat Rafiuddin, Rafae Abdullah
Institution: Oklahoma State University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166884255