Work

Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts

Year: 2026

Type: conference-paper

Abstract: Despite the success of large language models (LLMs) in a wide range of applications, it has been shown that their performance varies across English dialects.Differences among English dialects are reflected in vocabulary, syntax, and writing style, and can adversely affect model performance.Several studies evaluate the dialect robustness of LLMs, yet research on enhancing their robustness to dialectal variation remains limited.In this paper, we propose two parameterefficient frameworks for improving dialectal robustness in LLMs: DialectFusion where we train separate LoRA layers for each dialect and apply different LoRA merging methods, and DialectMoE which is built on top of Mixture of Experts LoRA and introduces multiple LoRAbased experts to the feed-forward layer to internally model the dialectal dependencies.Our comprehensive analysis on five open-source LLMs for sentiment and sarcasm tasks in zeroand few-shot settings shows that our proposed approaches enhance the dialect robustness of LLMs and outperforms instruct and LoRA finetuning based approaches.

Source: Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)

DOI: 10.18653/v1/2026.vardial-1.24

Authors: Sanjh Maheshwari, Aniket Singh Rajpoot, Oana Cocarascu, Mamta .

Institution: University College London

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Engineering and Physical Sciences Research Council

Award: A framework for evaluating and explaining the robustness of NLP models