ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
Year: 2026
Type: conference-paper
Abstract: Reliable tool use is a fundamental capability for agentic systems, yet existing evaluations primarily focus on successful tool execution and provide limited insight into how agents recover from tool failures.To address this gap, we introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and fault-conditioned recovery.ToolMisuseBench evaluates recovery behavior across domains, fault types, and difficulty levels, using output-grounded success criteria and typed recoverability metadata.The seed-42 release includes 6,000 training, 900 development, and 1,200 public-test tasks covering CRUD, retrieval, file operations, and scheduling, including failures such as timeouts, rate limits, hidden first-call schema perturbations, authorization errors, and adversarial responses.Using the benchmark, we evaluate deterministic reference agents together with local 3-4B tool-calling LLMs (Qwen3-4B, Ministral-3B, and Llama-3.2-3B).On the public-test split, Parser+Repair achieves 0.767 task success and 0.736 conditional recovery success, compared with 0.158 task success when recovery is disabled.On a shared stratified 216-task diagnostic panel, Qwen3-4B, Ministral-3B, and Llama-3.2-3Bachieve task success rates of 0.796, 0.903, and 0.773, respectively.Fault-level analysis reveals distinct recovery weaknesses that overall task success alone fails to capture.ToolMisuseBench provides a controlled and reproducible framework for diagnosing tooluse recovery under frozen fault plans, enabling more rigorous evaluation of robust tool-using agents beyond execution accuracy.
Authors: Rista Baral, Akshey Sigdel
Institution: University of Delaware
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7149873390