Year: 2025
Type: preprint
Abstract: With the increasing deployment of language models in edge devices and resource-constrained environments, the demand for highly capable small language models (SLMs) has grown significantly. Although recent large-language models (LLMs) have demonstrated strong performance in math and reasoning tasks, there is limited benchmarking of models under 3 billion parameters on similar challenges. This paper presents a targeted evaluation of 13 SLMs, each with equal to or fewer than 3B parameters, on a curated set of 50 math word problems spanning three difficulty levels: easy, medium, and hard. The models were evaluated using a consistent explanation-style prompting strategy. Models such as phi-2, Qwen1.7B, and Qwen2.5B-Math achieved 92% overall accuracy, demonstrating robust reasoning capabilities even on hard questions. In contrast, smaller or less specialized models such as MobileLLaMA-2 and OpenELM-3B showed much lower accuracy, particularly on harder tasks. These results suggest that well-trained and domain-tuned SLMs can match or even surpass general-purpose models of similar or larger size. The findings highlight the importance of math-centric training and careful architecture selection in improving reasoning performance in compact models.
Author: Harshit Shah
Institution: Birla Institute of Technology and Science, Pilani
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4413515624