What Moves the Pareto Frontier in Tool-Using Agents? A Compute-Aware Study of ReAct Variants
Year: 2026
Type: conference-paper
Abstract: Tool-using LLM agents are typically compared by accuracy alone, despite deployments being constrained by inference cost.We present a budgeted evaluation of common strategies for improving ReAct-style tool agents (multisample aggregation, iterative self-correction, and post-hoc answer revision) using Pareto analysis of cumulative accuracy versus token budget on three benchmarks (HotPotQA, FEVER, GSM8K) with Gemini 2.5 Flash.All experiments use three random seeds (N =500 per seed for HotPotQA/FEVER; N =1,015 for GSM8K); budgeted curves are computed post hoc from per-instance token logs.In our offline evaluation, Reflexion attains the highest accuracy on tool-heavy benchmarks (HotPotQA, FEVER), while CoT-SC leads on GSM8K.Reflexion's reported token costs are optimistic lower bounds because retries are stopped using ground-truth feedback, and its accuracy is similarly optimistic-a deployment without access to ground-truth labels would not achieve the same accuracy because the goldlabel stopping criterion would be unavailable; both costs and accuracy would differ in practice.Sampling-based approaches often spend 3-5× more tokens for comparatively small gains on tool-heavy tasks.GSM8K, a pure-math benchmark with minimal tool interaction, shows substantially larger gains for CoT-SC, TCAR, and Reflexion-larger than on tool-heavy benchmarks, though less sharply separated than headline accuracy alone would suggest-consistent with repeated tool trajectories being an important contributor to the observed efficiency gap in our tool-heavy settings.We provide a compute-aware evaluation protocol (frontier analysis and marginal-cost metrics) and practical guidance for choosing agent designs under different budget regimes.
Author: Rishi N. Simhadri
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166800698