Work

From Atomic to Agentic: Towards Interpretable Evaluation of LLMs’ Agentic Mathematical Capabilities

Year: 2026

Type: conference-paper

Abstract: Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence.However, most existing math benchmarks evaluate only final answers.This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents.To bridge this gap, we present a process-level benchmark 1 designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs.Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities.We design a comprehensive suite of planning, action, and feedback tasks across both textual and multimodal contexts, supported by an automated pipeline that synthesizes high-quality trajectories and produces fine-grained annotations via controlled LLM rewriting.Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles.This demonstrates that processlevel evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.* Work done during the internship at Tencent Youtu Lab.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/2026.emnlp-main.194