Work

Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs

Year: 2026
Type: preprint
Abstract: Preprint and complete reproduction archive for "Dialect-Level Generalization of Arabic Sentiment Classifiers: An Exact-Duplicate-Free, Uncertainty-Aware Evaluation of Encoders and LLMs." AbstractHow well do Arabic sentiment classifiers generalize across dialects, and how should this be measured? We call a classifier robust across dialects if its macro-F1 is both high and even, summarized by harmonic-mean (HM-F1) and worst-dialect F1 with hierarchical bootstrap intervals over tweets and seeds. We evaluate 11 fine-tuned encoders and four instruction-tuned LLMs on a deduplicated, leak-free version of ArSarcasm (MSA, Egyptian, Levantine and Gulf Arabic) and replicate the transfer experiments on ArSarcasm-v2. The official ArSarcasm splits share 44 tweets between training and test, but the resulting inflation is small (-0.005 to +0.004 HM-F1 across four core encoders). Under pooled training MARBERTv2 is best (HM-F1 0.870), but differences among the 7 strongest Arabic encoders, and to the best 8-shot LLM (Fanar-1-9B-Instruct, 0.849), lie within uncertainty, and Levantine is the hardest dialect for every encoder. With training size matched, leaving the target dialect out or training on MSA only costs Arabic encoders no detectable HM-F1: losses above about 0.045 are excluded, although single-dialect effects below roughly 0.07-0.10 cannot be resolved, and adding in-dialect tweets yields no statistically detectable gain. This replicates on ArSarcasm-v2; the leave-one-dialect-out loss of the multilingual XLM-R does not. With tokenizer fixed, dialect-only and classical pretraining are less robust than mixed or MSA pretraining. A variance-based robustness index, on accuracy or F1, ranks a majority classifier among the most robust of 79 systems and should not be used. Automatic dialect identification agrees with only 0.56 of the gold dialect labels, yet conclusions and encoder tiers are unchanged on the agreeing subset. Archive Contents- main.pdf: Full 24-page preprint manuscript (CC BY 4.0).- arabic-dialect-robustness-zenodo.zip: Complete reproduction archive containing LaTeX source files, all 21 tables and 5 vector figures, sanitized split metadata (internal row indices, split assignments and labels with raw tweet text and Twitter/X identifiers removed), multi-seed prediction arrays and Python verification scripts (MIT License for code; CC BY 4.0 for derived split metadata).- Code Mirror: https://github.com/Hamza-Nayab/arabic-dialect-robustness
Language: English
Open Access status: green
OpenAlex ID: W7235678900
Cites:
Cited by:
Related to: 10