Get started
Work

Writing rules for agent memory with two extractor models: a paired extension

Year: 2026
Type: preprint
Abstract: A research note on how much a writing criterion for agent memory depends on the model that extracts the memories. An agent memory system turns past conversations into short memories with a language model, guided by a prompt that says what to keep. In an earlier study on LongMemEval, a rule-based criterion (C2) beat the deployed one (C0) by +24.8 points of answer accuracy with gemini-2.5-flash-lite as the extractor. This study ran C0, C2 and a third prompt (P), which drops C0's lists of what to keep and states what the memories are for, with gemini-2.5-flash-lite and with gemini-2.5-flash (thinking off), on the same 242 questions. Every memory, answer and grade was remade, under a public pre-registration frozen before any paid call of the run and reviewed beforehand by two models of other families. Both controls passed. The pre-registered interaction, the change in the C2 advantage between the two extractors, was −4.1 points (95% interval −12.3 to +4.1; second judge −5.0, −13.4 to +3.5): inconclusive, which is not evidence that the advantages are equal. As secondary contrasts, C2 beat C0 by +21.1 points with flash (+14.8 to +27.4) and by +25.2 with flash-lite; P beat C0 with both extractors and scored 3.7 points below C2 with flash (−9.4 to +1.9). The comparison is between two configured models under one reader, on questions the earlier study had already used; it does not isolate extractor strength. The archive carries the note, its provenance file, the pre-registration, the runner, analysis and tests, the cost smokes, the run's rows, receipt and analysis, the four adversarial reviews, and the earlier study's modules the runner imports. The benchmark is public and is not redistributed.
DOI: 10.5281/zenodo.24057927
Language: English
Open Access status: green
OpenAlex ID: W7225581894
Cites:
Cited by:
Related to: 10