Evaluating Large Language Models for Medical Named Entity Recognition in Urdu: A Benchmark Study
Year: 2026
Type: conference-paper
Abstract: Medical named entity recognition (NER) is a crucial task in natural language processing (NLP) for extracting meaningful entities such as diseases, symptoms, medications, body parts, and treatments from clinical text.However, NER in low-resource languages like Urdu remains underexplored due to limited annotated datasets.In this study, we evaluated the performance of two state-of-the-art large language models (LLMs), ChatGPT-4o and LLAMA 3.2, on Urdu medical NER using a dataset of 2,057 health-related Urdu news headlines manually annotated across five entity categories.Both models were evaluated using precision, recall, and F1-score.It was found that both models exhibited low precision and moderate recall.ChatGPT-4o achieved the highest F1 for Disease (0.35) while LLAMA 3.2 reached slightly lower F1 scores for Disease (0.33).Both models performed poorly on treatmentrelated terms, with F1 scores of 0.036 (LLAMA 3.2) and 0.011 (ChatGPT-4o).Micro-average F1-scores were 0.187 for ChatGPT-4o and 0.183 for LLAMA 3.2, indicating comparable overall performance.These findings highlight the challenges of medical NER in low-resource languages and underscore the need for domain-specific fine-tuning, transfer learning, few-shot learning, and prompt engineering to improve performance.
Institutions: Dow University of Health Sciences, Aga Khan University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166536125