Year: 2026
Type: conference-paper
Abstract: Automatic Speech Recognition (ASR) evaluation has traditionally relied on Word Error Rate (WER), a metric that treats all errors equally and obscures critical failure modes.In this paper, we present a fine -grained human evaluation of Meta's recently released OmniASR system on Saudi Arabic dialects using the SADA dataset.Three trained annotators evaluated 103 audio samples, producing 264 annotations across two dimensions (comprehensibility and naturalness) while categorizing errors using a novel 10 -category Arabic-specific error taxonomy.OmniASR achieved a mean WER of 42.2% and mean comprehensibility of 3.62/5, but exhibited a polarized performance pattern: 32.6% of transcriptions achieved perfect scores while 21.2% were essentially unusable.Error analysis reveals that hallucinations and deletions have the greatest negative impact on comprehensibility (-1.64 and -1.57points respectively), roughly 6× more damaging than named entity errors.WER correlates with human comprehensibility ratings at a level comparable to inter-annotator agreement (r = -0.679 vs. pairwise annotator r = 0.61-0.67),but lacks the diagnostic granularity to reveal which error types drive quality degradation.These findings motivate the use of error-type-aware evaluation frameworks that complement WER with fine-grained analysis for Arabic ASR systems.
Author: Hend Al-Khalifa
Institution: King Saud University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166571686