Work

The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior

Year: 2026

Type: preprint

Abstract: As large language models continue to improve, agentic systems are becoming increasingly important, and tools are a key design dimension because they determine how agents access information and take action in their environments. Prior work on agent tooling has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model. We refer to this latter design dimension as tool architecture. We study tool architecture in coding agents through controlled experiments on repository-level issue fixing, comparing six tool architectures that hold the underlying information and actions similar while varying how they are organized and exposed to the model, across three actors and a total of 11,700 trajectories. Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior: Compared to a basic architecture where the agent has only the bash tool, more structured low-level interfaces improve consistency across repeated attempts by up to 4.7 $\times$; natural-language search broadens repository exploration and increases access to relevant files by more than 11%; and Python CodeAct-style interfaces achieve similar task performance with 41.6% fewer steps and 56.3% lower token usage. By contrast, lightweight text-based cognitive-scaffolding tools, such as tools that let the agent record intermediate reasoning, have limited effect on actor behavior.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2608.11386

Authors: Xu, Xiangzhe, Saghir, Hamidreza, Wu, Qianhui, Côté, Marc-Alexandre, Wang, Tong, Lakkaraju, Kiran, Pei, Kexin, Zhang, Xiangyu

Institutions: Purdue University West Lafayette, Microsoft (United States), University of Chicago

Topic: Software Engineering Research

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Defense Advanced Research Projects Agency, Office of Naval Research

Awards: SHF: Small: AI Model Debugging by Analyzing Model Internals with Python Program Analysis, G4353487559, Proto-OKN Theme 1: Knowledge Graph Construction for Resilient, Trustworthy, and Secure Software Supply Chains, G6169757564, SLES: Whitebox Testing, Debugging, and Repairing for Multi-module Autonomous Vehicles in Near-Collision Traffic Scenarios, SHF: Medium: Principled Co-Reasoning of Software and Natural-Language Artifacts, G8595151890, G8708970860, G8876996369

OpenAlex ID: W7202342863

Cites:

Cited by:

Related to: