Benchmarking Pre-trained Open-source Large Language Models for Uzbek: Evaluating Performance in a Low-resource Setting Across Translation, Comprehension, and Generation
Year: 2025
Type: article
Abstract: Uzbek is a Turkic language spoken by more than 60 million people, yet it is still underrepresented in natural language processing because of scarce resources and technical support. In this work, we tested seven open-source language models with 0.6–2 billion parameters—DeepSeek-R1-Distill-Qwen-1.5B, Falcon3-1B-Instruct, Gemma2-2B-it, GLM-Edge-1.5B-Chat, Kimi-K2-Instruct, Llama-3.2-1B-Instruct, and Qwen3-0.6B—on Uzbek tasks such as translation (Uzbek↔English), reading comprehension, and text generation. We built a dataset of 180 samples covering formal, news, technical, conversational, and jargon texts, and ran zero-shot evaluations on a RunPod A40 GPU using Hugging Face Transformers. The results show that Kimi-K2 Instruct is the only model reaching practical usability, scoring an average BLEU of 0.54 for translation while also performing reliably in comprehension and generation. The other models struggled, often producing incoherent output or mixing languages. Overall, these results underline the difficulty of using small-scale LLMs for low-resource languages. Among the tested models, Kimi-K2-Instruct stands out as the best foundation for further fine-tuning, especially in Uzbekistan’s limited compute setting. To encourage more research, we have released the dataset, code, and results openly.
Author: Khasankhon Yusupkhujaev
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7083674325
FWCI:
Cites:
Cited by:
Related to: