Comprehension, Not Only Compliance: How Denominators and Detectors Shape Cross-Lingual Safety Gaps for Urdu, Roman Urdu and Code-Switched Prompts
Year: 2026
Type: preprint
Abstract: Multilingual safety evaluations usually summarise a model by a single “safety gap” betweenEnglish and another language. We re-analyse a released evaluation, VeriAudit v0.1: two open-weight 8B models (Qwen3-8B, Aya Expanse 8B), 30 harmful requests, each written in English,Urdu script, Roman Urdu and Urdu–English code-switching, and 240 labelled responses. Wefind that the gap depends on two choices that are rarely reported. First, the denominator.Excluding non-scoreable outputs (loops, garbled text, “I don’t understand”) inflates the gapwhere models fail to comprehend. Counting only actual harmful compliance on all 30 pairs,Qwen3-8B’s Roman Urdu gap falls from the published 39.5 to 26.7 percentage points, and itis no longer significant after Holm correction (p=.086). Treating the same outputs as unsaferaises it to 56.7 points. Second, the detector. The evaluation’s English refusal heuristic found 0%of Urdu-script refusals and 16–19% of Roman Urdu and code-switched refusals, against 92% inEnglish. On the same responses it would have reported gaps of 42.5–70 points. We reproduce all191 published numbers of v0.1 exactly from raw outputs. An exploratory follow-up adds a newermodel (Qwen3.5-9B), three sampled responses per prompt and two independent automatedjudges. It reproduces the response-language drift and the direction of the largest gaps, andshows that the judges agree on harmfulness far more often than on the type of refusal. Weargue that cross-lingual safety gaps for these forms must be reported with their comprehensioncomponent and with a validated multilingual measurement, and we release the reproductioncode.
Source: Figshare
Author: Abeera Najam
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: Urdu language, Roman Urdu, code-switching, Qwen3-8B
Open Access status: green
OpenAlex ID: W7236651703
