Year: 2024
Type: preprint
Abstract: Audit kit for the memory-retrieval benchmark of Mnemosyne OS, a local-firstmemory system for AI assistants. This archive contains everything needed to audit the published scoreindependently: the per-question ledger for every run, the scoring andverification scripts, the methodology, and the raw run logs. Headline result: 72.9% on LongMemEval-M, full-haystack variant. It isreported as a LOWER BOUND, not a leaderboard score — full-haystack is theharder setting and is rarely the one reported. The kit exists so the numbercan be audited rather than trusted: verify.js re-runs the scoring over theraw ledger. Contents:- verification-kit/results/*.jsonl — per-question ledgers (48q baseline, 8q engine multi-session, 12q local sovereign)- verification-kit/verify.js and scoring.js — scoring and verification- verification-kit/METHODOLOGY.md — protocol, and what the result does not claim- fullhaystack-2026-07/ — raw run logs and campaign summary Licensing: documentation and data under CC BY 4.0; code under MIT.
Source: arXiv (Cornell University)
Authors: Wu, Di, Wang, Hongwei, Yu, Wenhao, Zhang, Yuwei, Chang, Kai-Wei, Yu, Dong
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4403571340