Work

Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting

Year: 2026

Type: article

Abstract: Large Language Models (LLMs) have advanced multilingual NLP significantly, yet their effectiveness for Bengali, a globally important but low-resource language, remains understudied due to limited linguistic resources and the absence of a standardized evaluation benchmark. This is due to the unavailability of large linguistic resources and the lack of a standard benchmarking framework, which has yet to be established. This study aims to address the underrepresentation of Bengali text classification by leading a systematic evaluation. It involves assessing multiple open-weight LLMs on Bengali text classification tasks and observing their limitations and regular error patterns. The procedure consists of six open-weight LLMs, which include LLaMA-3.2-3B, Mistral-V3-7B, DeepSeek-R1-8B, Phi-4-14B, Gemma-2-27B, and Qwen-2.5-72B, and are evaluated using four critical text classification tasks in Bengali, namely sentiment analysis, emotion recognition, hate-speech detection, and fake-news classification. Furthermore, two prompting methods are compared in the context of zero-shot and few-shot prompting using both the vLLM inference engine and Hugging Face Transformers pipelines to investigate the adaptability of the models in resource-limited settings. Across all four tasks, the two largest models (Qwen-2.5-72B and Gemma-2-27B) obtained the highest scores, although the ranking was not strictly monotonic in parameter count and, because each scale is represented by a single model family, the design cannot attribute these differences to model size alone. When training in zero-shot, Qwen-2.5-72B showed the best performance at 79.90% accuracy and 79.88% Macro-F1 score, showing amazing accuracy on hate-speech detection. In the case of fake news, Gemma-2-27B performed best at 84.45% accuracy and 84.33% Macro-F1 score. With a few-shot set-up, Gemma-2-27B and Qwen-2.5-72B demonstrated a high level of performance in terms of Macro-F1 score, reaching 84.17% for fake-news detection and 66.80% for emotion recognition, respectively. Paired multi-model tests (Cochran's Q with Holm-corrected McNemar post-hoc comparisons) confirmed significant differences among the six models across all tasks and prompting settings, whereas the effect of five-shot prompting, obtained with a single fixed demonstration set, was significant for only 4 of 24 model-task combinations after correction and never for the three largest models. The findings characterise benchmark classification performance under the evaluated settings rather than real-world effectiveness, and indicate that model scale, multilingual pre-training, and in-context demonstrations are jointly associated with, but cannot be causally isolated as drivers of, performance on Bengali text classification.

Source: PLoS ONE

DOI: 10.1371/journal.pone.0361368