Roman Urdu Input Costs Small Local ReAct Agents up to 35 Accuracy Points: A Paired Benchmark on Clinic Questions
Year: 2026
Type: preprint
Abstract: Tool-using agents built on small language models can run on a clinic's own computer, but their users in Pakistan and in Pakistani communities in the UK often type in Roman Urdu: Urdu written in Latin script, without a fixed spelling. We ask how much accuracy a ReAct agent loses on Roman Urdu, which words cause the loss, and whether two cheap fixes help. We built 96 questions over a fictional physiotherapy clinic from 12 templates, each in English, common Roman Urdu spelling and chat spelling, with gold answers computed from the records. We ran four local models (Qwen2.5 3B and 7B, Llama 3.2 3B, Llama 3.1 8B) at temperature 0, for 3,838 scored calls. With an English worked example, ReAct accuracy fell from English to Roman Urdu by: 34.7 points for Qwen2.5 3B 31.2 points for Llama 3.2 3B 21.9 points for Llama 3.1 8B All three 95% bootstrap intervals are below zero. Qwen2.5 7B showed no loss (+1.0). Chat spelling cost Qwen2.5 3B a further 22.1 points. The largest losses were on time words (saal baad, saal pehle), mila kar (combined) and clinic days. For the two Llama models, answering with the whole database in the prompt showed no Roman Urdu gap, so the loss appeared inside the agent loop. A Roman Urdu worked example and translating the question first helped Llama 3.2 3B (+16.7 and +20.8 points) but not Qwen2.5 3B. A pilot also showed that sending the ReAct transcript as one user message, instead of separate chat turns, dropped Llama 3.2 3B to 7% in English. Code, items and every call with its transcript are public.
Author: Hussain, Muzammal
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225460702