WhispAdapt: Multilingual dialect recognition in low-resource Dravidian languages
Year: 2025
Type: article
Abstract: Dialect variation in pronunciation, vocabulary, grammar, and syntax poses a persistent challenge for multilingual speech systems. This challenge is especially noticeable in low-resource Dravidian languages. Existing models often struggle due to the lack of annotated dialectal corpora and poor adaptability to regional speech patterns. To tackle this gap, we propose WhispAdapt, a lightweight and efficient adaptation of the Whisper encoder built for dialect classification tasks. Our framework applies parameter-efficient fine-tuning methods such as Low-Rank Adaptation (LoRA) and its quantized variant (QLoRA), allowing the model to learn effectively even with fewer samples while keeping computation and memory demands low. We made a 65.9-hour dialectal speech dataset with twelve dialects from Tamil, Kannada, and Malayalam for evaluation. The experimental results demonstrate that WhispAdapt-LoRA achieves impressive results compared to the other variant-WhispAdapt-QLoRA. It achieved macro F1-scores of 93% for Kannada, 97% for Malayalam, and 90% for Tamil—surpassing benchmarks such as Wav2Vec2, HuBERT, and WavLM by as much as 25%. Moreover, WhispAdapt also generalizes effectively to unseen accents on the Accent_DB benchmark. Overall, this study offers a scalable step toward improving dialect modeling for other underrepresented languages.
Source: Array
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,680
OpenAlex ID: W4416931105