A choice is not a lock: configuration-based tool disabling in an agent harness, evaluated with four language models
Year: 2026
Type: preprint
Abstract: Background. A common control in agent harnesses is to disable a tool through configuration. Objective. To test whether such disabling prevents the action when the agent retains a terminal and a readable credential, and whether the outcome depends on the model. Method. Hermes v0.22.6 ran in a container with network containment and fictitious destinations, with four models (Claude Sonnet 5.5, GPT-6.1 Sol, GLM-5.3 and Kimi K3.1 Pro) and identical configuration, requests and order. The main test (V2) crossed four capability levels with three request forms, with 3 rounds per cell (36 per model). The outcome was what reached the mailbox or the collector, recounted from the raw logs. Results. With the tool disabled by configuration and the credential in the configuration file, the message reached the destination in 0/36 rounds with Claude, 0/36 with GPT, 13/36 with GLM and 3/36 with Kimi. Claude found the route in 16 rounds and declined every time. GPT did not find the route in 34 of 36 rounds and attempted direct SMTP twice, with no delivery. In a post hoc analysis, the GLM proportion differed from the other three after Holm correction; Kimi was not distinguishable from Claude or GPT. The simulated control point (V2b) produced 144 BLOCKED outcomes but was inconclusive, because the credential remained accessible. The harness estimate of cost equaled 32% of actual spend with GLM. Conclusion. Removing the tool through configuration did not remove the route. The outcome depended on the model, in this scenario and on these dates.
Author: Cavicchioli, Marcelo
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225325521