Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) achieve impressive performance on complex mathematical benchmarks yet sometimes fail on basic math reasoning while generating unnecessarily verbose responses.In this paper, we present LLMTHINKBENCH, a systematic benchmark and comprehensive empirical study to evaluate the efficiency of reasoning in LLMs, focusing on the fundamental tradeoff between accuracy and overthinking.First, we formalize the accuracy-verbosity tradeoff.Second, we introduce the Overthinking Score, a harmonic-mean metric combining accuracy and token-efficiency for holistic model evaluation.Third, we establish an evaluation protocol with dynamically-generated data across 14 basic math tasks.Fourth, we conduct a large-scale empirical study evaluating 53 LLMs, including reasoning and quantized variants across different reasoning budgets.Fifth, we release LLM-THINKBENCH as an open-source Python package and public leaderboard for reproducibility.Our findings reveal: 1) model performance on complex benchmarks does not translate directly to basic math reasoning; 2) reasoning models generate ∼18× more tokens while sometimes achieving lower accuracy and exhibit catastrophic collapse when tokens are constrained, dropping by up to ∼36%; 3) the accuracyverbosity relationship is non-monotonic with extended reasoning budgets yielding diminishing returns (GPT-5/o-series models show zero accuracy gain from low → medium → high reasoning effort).Our findings challenge the assumption that longer reasoning in LLMs necessarily improves mathematical reasoning.0 .5 B 0 .6 B 1 B 1 B 1 B 3 B 4 B 7 B 8 B 1 4 B 3 2 B 7 0 B 7 2 B Model Parameters (B) 0 20 40 60 80 100 Accuracy (%) (a) Model Family Scaling Comparison Llama Qwen2.5 Qwen3 0 .6 B 1 .7 B 4 B 8 B 1 4 B 3 2 B Model Parameters 0 20 40 60 80 100 Accuracy (%) (b) Token Budget Impact Full Token Budget Constrained (1024 tokens) 50.0 70.2 81.9 82.1 86.5 84.1 27.0 30.7 31.7 28.4 35.2 30.2 0 .5 B 1 .5 B 3 B 7 B 1 4 B 3 2 B 7 2 B Model Parameters 0 20 40 60 80 100 Accuracy (%) (c) Quantization Impact FP16 (Original) GPTQ 8-bit GPTQ 4-bit
Authors: Gaurav Srivastava, Aafiya Shamshad Hussain, Sriram Srinivasan, Xuan Wang
Institution: Virginia Tech
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4415314084