Year: 2023
Type: conference-paper
Abstract: Linguistic variability poses a challenge to many modern ASR systems, particularly Dialectical Arabic (DA) ASR systems dealing with lowresource dialects and resulting morphological and orthographic variations in text and speech.Traditional evaluation metrics such as the word error rate (WER) inadequately capture these complexities, leading to an incomplete assessment of DA ASR performance.We propose AraDiaWER, an ASR evaluation metric for Dialectical Arabic (DA) speech recognition systems, focused on the Egyptian dialect.AraDi-aWER uses language model embeddings for the syntactic and semantic aspects of ASR errors to identify their root cause, not captured by traditional WER.MiniLM generates the semantic score, capturing contextual differences between reference and predicted transcripts.CAMeLBERT-Mix assigns morphological and lexical tags using a fuzzy matching algorithm to calculate the syntactic score.Our experiments validate the effectiveness of Ara-DiaWER.By incorporating language model embeddings, AraDiaWER enables a more interpretable evaluation, allowing us to improve DA ASR systems.We position the proposed metric as a complementary tool to WER, capturing syntactic and semantic features not represented by WER.Additionally, we use UMAP analysis to observe the quality of ASR embeddings in the proposed evaluation framework.
Authors: Abdulwahab Sahyoun, Shady Shehata
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4386566596