Roman Urdu Breaks Dense Retrieval: Script, Spelling Variation and a Transliteration Fix
Year: 2026
Type: preprint
Abstract: Much everyday Urdu is typed in the Latin script (Roman Urdu), with no standard spelling, while the text people want to search is written in Urdu script. We measure what this does to retrieval. On a new set of 200 test questions over 3,674 Wikipedia passages, each question asked in Urdu script, in English and in three Roman Urdu spellings, every dense embedder we test loses between 29 and 59 points of Recall@5 when the question moves from Urdu script to Roman Urdu, and BM25 finds almost nothing. A second test with human-written Roman Urdu questions from Belebele shows a similar gap (40 to 51 points), although there the Roman Urdu is also worded differently. Spelling variation is a smaller effect than script, but for up to 14% of questions one spelling finds the answer and another misses it. Rule-based spelling normalization makes things worse. Transliterating the query into Urdu script before searching closes most of the gap: with an LLM transliterator the best embedders come within a few points of their Urdu-script score, and a small open transliteration model recovers 90% of the LLM's gain without an API call, while a general 7B open LLM does markedly worse. We release the question set and code: https://github.com/masooddahmedd/roman-urdu-retrieval
Author: Ahmed, Masood
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7237205327
