HierMem: Context Curation Over Context Scaling — Hierarchical Memory with Invariant Constraint Placement for Long-Horizon LLM Conversations
Year: 2026
Type: preprint
Abstract: The industry assumes that larger context windows are the solution to long-horizon conversational AI. This assumption is fundamentally flawed. Flooding a Large Language Model (LLM) with raw historical turns actively degrades reasoning through the "lost in the middle" effect, dilutes critical user constraints, and incurs massive compute costs. LLMs do not need to process every past interaction; they require only highly relevant information, curated intelligently like human recall. I present HierMem, an OS-inspired systems architecture that explicitly challenges the full-history paradigm. HierMem replaces brute-force context dumping with a four-level hierarchical memory (L0-L3), an invariant non-evictable constraint zone, and a lightweight stateless curator model. By separating memory orchestration from response generation, constraint salience is guaranteed by placement policy, while topical relevance is handled by precision routing. Evaluated on 15 synthetic benchmark conversations (50 turns each) across software engineering, finance, legal, and healthcare domains, HierMem achieves a 4.7x architectural compression ratio. In the finalized Architecture C run (Qwen2.5-14B main model), HierMem obtains the best aggregate judge score (8.461) and a 93.3% constraint survival rate. Relative to Raw LLM, HierMem improves quality by +1.553 judge points while reducing average session compute cost by 36.9%. This work demonstrates that structured curation, rather than context scaling, is the optimal path for reliable long-horizon assistants.
Source: Research Square
Author: Yash Doke
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7164894427
