Work

What Is in a WER? Reproducible Evaluation of Whisper for Sudanese Arabic

Year: 2026
Type: preprint
Abstract: Published word error rates (WER) for dialectal Arabic speech recognition are hard to compare, because text normalisation, decoding settings and data splits often go unreported. For practitioners who choose a model for real users, this matters. We re-evaluate Whisper on Sudanese Arabic with a fully specified protocol. (1) An independent run of a public finetuned checkpoint reproduces its reported score (67.38 vs. 67.7 WER). (2) Arabic orthographic normalisation (alef, ya, ta marbuta and the conjunction wa) lowers cross-corpus WER by 10.8–12.7 points for models fine-tuned on the open Sudanese corpus, against 4.9–7.1 points for zero-shot models. Much of the apparent cross-corpus error is spelling convention, not recognition. (3) Zero-shot Whisper large-v3 matches the fine-tuned small model on WER, though not on CER. (4) On a new speakerdisjoint split, LoRA adaptation helps unseen speakers of the same corpus but brings no significant gain on a second Sudanese test set. (5) A widely used mirror of the corpus pairs audio with the wrong transcripts: only 85 of 3,547 rows are aligned. We release the split, the normaliser, the scoring code and an alignment checker, and we turn these findings into an evaluation checklist for low-resource dialect ASR.
Language: English
Open Access status: green
OpenAlex ID: W7236039403
Cites: 2
Cited by:
Related to: 10