Year: 2026
Type: conference-paper
Abstract: Enhancing the reasoning capabilities of Large Language Models (LLMs) is a key strategy for building Agents that "think then act."However, recent observations, like OpenAI's o3, suggest a paradox: stronger reasoning often coincides with increased hallucination, yet no prior work has systematically examined whether reasoning enhancement itself causes tool hallucination.To address this gap, we pose the central question: Does strengthening reasoning increase tool hallucination of LLM Agents?We address this gap by introducing SIMPLETOOL-HALLUBENCH, a diagnostic benchmark measuring tool hallucination.Through controlled experiments, we show that across RL, distillation, and toggleable reasoning modes, gains in task performance are consistently accompanied by higher tool hallucination rates.This effect is training method-agnostic and transcends simple overfitting, as training even on non-tool-related tasks (e.g., mathematics) still amplifies tool hallucination.Controlled ablations further reveal that the reasoning itself, rather than RL training in general, is most closely associated with the hallucination increase.We evaluate mitigation strategies including Prompt Engineering and Direct Preference Optimization (DPO), revealing a fundamental reliability-capability trade-off: reducing hallucination unavoidably degrades utility.Our findings demonstrate that under current reasoning enhancement methods, improved reasoning is systematically associated with increased tool hallucination, highlighting the need for training objectives that jointly optimize capability and reliability.Our
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166868746