Zero-Shot Performance of Compact Whisper Variants on Urdu-English Code-Mixed Speech: An Empirical Investigation Using WER and Switch Point Analysis
Year: 2026
Type: preprint
Abstract: This study presents the first systematic zero-shot evaluation of compact OpenAI Whisper variants, Tiny (39M parameters), Base (74M), and Small (244M), on Urdu-English code-mixed speech, a linguistic phenomenon ubiquitous among educated urban Pakistani speakers yet largely unaddressed in automatic speech recognition literature. A purpose-built corpus of 327 recordings from 11 bilingual speakers was constructed, stratified into three tiers of mixing intensity: mild, moderate, and heavy. Nine experimental conditions were evaluated by crossing three model sizes with three language inference configurations: Urdu-forced, English-forced, and automatic detection. Recognition quality was assessed using Word Error Rate (WER) alongside Switch Point Error Rate (SPER), a diagnostic metric formalized in this work to isolate transcription failures at language-transition boundaries. Results demonstrate that recognition error increases monotonically with mixing intensity, with up to 36.6% WER inflation from the mildest to the heaviest tier. SPER exceeds 0.93 across all nine conditions universally, revealing that language boundaries constitute systematic failure zones rather than incidental error sites, a structural deficit attributable to monolingual pre-training rather than insufficient model capacity. The Small variant paired with English-forced inference achieves the lowest error rate (WER = 0.899) and the highest consistency (CV = 10.72%), making it the recommended configuration for practical deployment. These findings provide quantitative evidence of where and why compact Whisper models fail with code-mixed South Asian speech, along with concrete deployment guidance for practitioners working in multilingual settings.
Author: Mohammed haroon, Abdullah Haroon
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7165040509
Cites:
Cited by:
Related to: