Year: 2026
Type: conference-paper
Abstract: Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models, yet why it works particularly well for code-mixed text remains unexplained.We propose that LoRA's efficiency stems from geometric structure in multilingual pre-trained models: code-mixed embeddings concentrate in low-dimensional cross-lingual subspaces.Through spectral analysis of mBERT and MuRIL on Hindi-English (Hinglish) data, we establish that pre-trained attention weights have effective ranks of 437-441, while LoRA updates (r = 4, 8, 16) exhibit ranks of 2.1-5.9-a136× average compression.Cross-lingual geometry measured via Centered Kernel Alignment (CKA) shows Hinglish embeddings align strongly with Hindi (CKA=0.279)but weakly with English (0.093), compared to a monolingual baseline of 0.074.Statistical tests (Wilcoxon p < 10 -19 ) and permutation ablations confirm these differences are robust.We interpret the convergence of geometric overlap (3.77× baseline) and empirical compression (136×) as evidence that low-rank adaptation exploits pre-existing multilingual structure.Findings are demonstrated on token-level language identification; extensions to other language pairs and tasks remain open questions.
Authors: Shashank Vishwakarma, Rakesh Kumar
Institution: Gautam Buddha University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166505374