MaskLoRA: Low-Rank Subspace–Induced Token Masking for Efficient and Faithful Language Models
Year: 2026
Type: conference-paper
Abstract: We propose MASKLORA, a plug-and-play masking mechanism that turns PEFT's lowrank subspace into a faithful, compute-aware token selector.Instead of relying on attention entropy or gradients, MASKLORA scores tokens by the LoRA-induced representation shift and learns differentiable gates with an objective combining task loss, expected L 0 sparsity, prediction consistency (KL), and a subspace concentration term.We provide a bound showing the KL between full and masked predictions is controlled by the sum of dropped scores, implying top-k selection minimizes the bound.Across encoders on GLUE and SQuAD, MASKLORA matches full-model accuracy while yielding 1.3-2.6×end-to-end speedups; on long-context NarrativeQA/GovReport it attains 1.5-3.6×with small but nonzero degradation (e.g., ≤ 0.4 at ρ=70%, up to ∼ 1.2 at ρ=50%, and up to ∼ 2.5 at ρ≈30%).Faithfulness improves over strong baselines (sufficiency 99.2%, necessity 59.3%, lower KL at ρ=50%), and energy falls accordingly (GPU Joules-per-correct 3.28 → 1.34 at ρ ≈ 30%).The method is KV-cache compatible, rank-and budget-controllable, and robust under OOD stress (HANS, adversarial sentiment, noisy SQuAD; at moderate budgets (ρ ≈ 50%) ≤ 0.9 abs.drop at matched budgets).Qualitative analyses show layer-wise retention concentrating at semantic bottlenecks and class-separated ∆h i embeddings.MASKLORA offers accuracymatched speedups with interpretable rationales and practical deployment hooks.hidden/attn streams h,α and LoRA Δθ ⇒ compute Δhi,ℓ,Δαi,ℓ.normalized, aggregated scores s^i ⇒ drive gate logits/selection.soft gates m~ (Hard-Concrete), budget ρ ⇒ loss terms use masked input.full forward f(x;θ+Δθ) ⇒ KL "teacher" logits.gradients ⇒ update LoRA params Δθ. gradients & calibration ⇒ update gate params ϕ, set threshold κ←F-1(1-ρ).scores s^i ⇒ hard mask m via threshold/top-k.hard mask m ⇒ kept-token set S. run f(x⊙m;θ+Δθ) with KVcache compaction ⇒ y^, logits, speedups.
Author: Rifat Rafiuddin
Institution: Oklahoma State University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140140792