Potion: When Ground Truth Holds No Ground, a Memory Measured on MEME
Year: 2026
Type: preprint
Abstract: We usually judge a memory by what it remembers. This paper judges one by what it stops saying. Potion keeps a dated record of the user's own words, written after each session by an open model and verified by code. Ripple, the part this paper measures, follows a change through everything that rested on it and marks what can no longer be stated instead of guessing. We ran it as a test on MEME (Jung et al., 2026): 78 held-out episodes that no one who built the system had read, a version frozen before they were opened, the benchmark's own judge and prompts, and predictions sealed before each run. One 27-billion-parameter open model, Qwen3.8-27B, wrote the record and read it on a single GPU and reached 0.611 pooled accuracy (0.584 to 0.639). The same reader scored 0.496 on the memory engine's context alone and 0.489 on the full transcript. The whole difference sits on the two tasks about what a memory should no longer claim. On Absence the reader was credited on 0 of 100 questions from the engine's context or the transcript and on 45 of 100 with Ripple's part; on Deletion, on 51 of 78 from the engine's context and on 72 of 78 with it. Where a new value has to be computed (Cascade), nothing we tested leaves the floor, ours included: 8 of 129. The paper also reports what the test taught us about the test: its rater credits a refusal and not a dated removal, and it states every dependency as an explicit condition, which none of the eleven dependencies in two records our agent sessions keyed by hand was. In this record: The paper (PDF). Per-question scores, 23 CSV files, one per reported number: the held-out set's four arms, the four arms without Ripple's part, the two controls, the twelve files of the three calibration rounds, and the pilot episode's rounds. Every reader's answers on the held-out set: one file per arm, ten arms, 926 rows each, released after the second exposure test (10 November 2026). Each row holds the question id, the round, the task, the hop where MEME gives one, a hash of the context the reader saw and the reader's answer. MEME's question texts and gold fields are not included; rows join the dataset by id at revision 8bd51aca. meme78_attribution_flags.csv: for each of the 307 questions after the change on Absence, Deletion and Cascade, whether the memory arm's context, and the same context without the legend, carried Ripple's line for the asked value, and whether each of the ten arms was credited. No context text. MANIFEST.csv, with each file's row count, mean and hash, and README.md, with the calibration rounds in full, the scores on our own stores, the source of every number in the paper, and the hashes of the sealed prediction files. Withheld: the contexts and the memory (Potion is closed), and every line of any conversation in our own stores. The benchmark: MEME, arXiv:2605.12477v1; dataset meme-benchmark/MEME, revision 8bd51aca. The memory engine beneath Ripple: the first preprint, doi:10.5281/zenodo.23016724.
Author: de Bruijn, Robin L.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225799751