Work

The memory the agent does not consult: a server instruction that works in evaluation and is rarely acted on in use

Year: 2026
Type: preprint
Abstract: A dated data note on whether a coding agent searches a memory server it is connected to. In our deployment, agents rarely searched the memory unless something told them to. The note tests three ways of telling a coding agent (Claude Code, claude-sonnet-5-5) when to search, and then checks whether the text that shipped changed ordinary use. On a frozen bank of 36 tasks whose answer depends on a memory fact the prompt does not name, the agent's pass rate with the memory server alone was 0.065. A skill file raised it by 0.250 (95% CI 0.139 to 0.370). A text the memory server sends when the client connects (the MCP server instructions, with rewritten tool descriptions) raised it by 0.491 (0.352 to 0.630) but failed a pre-registered limit on false triggers, on tasks the memory does not cover. A revision of that text that states what the memory covers and keeps its triggers inside that scope kept the gain (+0.463, 0.315 to 0.611) and brought false triggers within the limit; the revision changed several sentences, so the result belongs to the revision as a whole. Before the main runs, a positive control found a ceiling set by the product: the search tool returned only the first 200 characters of each hit, which release 3.8.0 fixed. No arm showed a measurable effect on writing to memory (S0 0 of 30, S1 1 of 30, T 0 of 30; T2 was not tested on writing). The arms were designed one after another on the same bank. A template of the scoped text shipped on 2026-11-08, configured for the owner's projects. In the owner's own sessions in the following week, 2 of 38 searched (0 of 269 before), which the pre-registered rule reads as no visible change: its threshold was not reached. The field reading counts searches in sessions whose need for memory is unknown, so it is not on the evaluation's scale. The operations agents on the same host, which never received the text, searched only during one weekly task whose answer needed the corpus. The note reports this contrast and does not explain it. The archive carries the note, its provenance file, the evaluation protocol, task bank and harness copied verbatim (the protocol holds the frozen-artifact hashes and every controlled-evaluation result), a dated recount of the two figures read on the production host, and the audits the note cites. The frozen evaluation database is not included and cannot be rebuilt in full from the public history, because four of its source files were later removed for privacy. The skill file, the run data with the analysis scripts that compute the effects from them, the real-use script and its reports, and the agent transcripts stay private, so the figures can be checked against the protocol but not recomputed from public material.
Language: English
Open Access status: green
OpenAlex ID: W7237213013
Cites:
Cited by:
Related to: 10