Get started
Work

Memory Without RAG: The Real Architecture

Year: 2026
Type: other
Abstract: Episode summary: Agent memory systems like mem0, Letta, Zep, and LangMem are built on fundamentally different architectures than retrieval-augmented generation — but the marketing language obscures what actually matters. This episode breaks down the real engineering decisions: how LLM-extracted fact stores differ from temporal knowledge graphs, why context-window-first approaches with external overflow change the game, and which pairings actually work in production. From mem0's deduplication pipeline to Letta's OS-inspired memory hierarchy and sleep-time compute, we examine the architectural divisions that define this space — and why the obvious answer of "just use RAG" falls short for stateful agents. Show Notes # Memory Without RAG: Understanding Stateful AI Frameworks When building agents that need to remember facts across conversations, the instinct is clear: use retrieval-augmented generation. Embed everything, store it in a vector database, pull relevant chunks at query time. It works for document retrieval. But agent memory is a different problem entirely. ## Why RAG Isn't Enough Standard RAG assumes a static corpus. You chunk it once, embed it, and retrieve at query time. But agent memory has to handle contradictions. When a user says "I live in New York" in session one and "I moved to Austin" in session three, a naive RAG system has both facts embedded. Depending on which scores higher on retrieval, you get the wrong answer. There's no concept of "this fact replaced that fact." There's another problem: OpenAI's own documentation discourages heavy RAG injection with reasoning models like o3. The models perform better with shorter, cleaner prompts. Stuffing more retrieved context in actually degrades performance on the most capable reasoning systems. The frameworks being built today solve this differently — by creating compact, accurate, temporally-aware representations of what an agent actually knows. ## Four Distinct Architectural Approaches The memory framework space divides into clear categories, each reflecting a different answer to where memory lives and how it gets formed. **LLM-Extracted Fact Stores (mem0)** mem0's pipeline is straightforward: every conversation passes through an LLM extraction step. The model identifies salient facts — discrete, human-readable claims. These get deduplicated, consolidated, embedded, and stored in a vector database. The strength is clean, human-readable memory. The cost is the extraction tax: every write involves an LLM call. At scale, that's latency and expense. The quality of memory is also bounded by extraction quality — if the model misses a nuance, that error persists. mem0 supports pluggable backends: Qdrant, pgvector, Chroma, Pinecone, Weaviate, Redis, MongoDB Atlas, Azure AI Search. There's an optional graph layer that extracts entities and relationships alongside facts, storing them in Neo4j, Memgraph, Kuzu, or Apache AGE (a PostgreSQL extension). The graph results supplement vector results at retrieval time without reordering them. **Temporal Knowledge Graphs (Zep)** Zep's Graphiti engine builds temporal knowledge graphs — storing not just facts but when they were learned, how they've evolved, and their relationships over time. This handles the contradiction problem natively: facts aren't replaced, they're timestamped and contextualized. **Context-Window-First with External Overflow (Letta)** Letta evolves the MemGPT paper's OS-inspired approach into a three-tier memory hierarchy. Tier one is core memory: structured XML-like blocks prepended to every prompt. The agent always has access to user name, preferences, project state — no retrieval needed, zero latency. The agent edits these with built-in tools: memory_rethink, memory_replace, memory_insert. Recommended limits are under fifty thousand characters per block, under twenty blocks. Tier two is archival memory: a general-purpose vector database for semantically searchable long-term storage. The agent uses explicit tools to search and insert. Tier three is recall memory: automatic, searchable conversation history. The most interesting innovation is shared memory blocks for multi-agent coordination. Multiple agents can attach to the same block. When one updates it, all see it immediately. This is a coordination primitive that requires no message queues or pub-sub infrastructure — shared state becomes the coordination mechanism. The latest advancement is sleep-time compute (April 2025 paper). Memory management and conversation handling were originally bundled in one agent, creating latency and reliability issues. Sleep-time architecture splits this: a fast primary agent handles real-time conversation using gpt-4o-mini, but cannot edit its own memory. An asynchronous sleep-time agent runs between conversations using a stronger model, with write access to memory blocks. It continuously consolidates and improves learned context. The numbers are striking: five-times reduction in test-time compute needed for the same accuracy on stateful reasoning benchmarks. Thirteen percent accuracy improvement on Stateful GSM-Symbolic, eighteen percent on Stateful AIME. Two and a half times cost reduction per query when amortizing sleep-time compute. The most recent addition: Context Repositories store memory blocks in git. Version control, branching, rollback of agent memory state. Memory as code. **Storage-Agnostic Libraries (LangMem)** LangMem provides memory management abstraction without committing to a specific storage backend or extraction strategy. It's a coordination layer rather than an opinionated system. ## The Benchmark Reality Check mem0's LOCOMO benchmark shows impressive numbers: twenty-six percent relative improvement over OpenAI Memory, ninety-one percent lower p95 latency versus full-context approaches, ninety percent fewer tokens. But every team benchmarks on metrics favoring their approach. The more interesting result comes from Letta's own research: their simple Filesystem abstraction — literally storing conversational histories in a file — scored seventy-four percent on LoCoMo. That beat several specialized memory tool libraries. For many use cases, the marginal benefit of sophisticated memory systems over well-organized file storage may be smaller than vendors claim. ## What This Means These aren't competing implementations of the same idea. They're fundamentally different answers to the memory problem. The choice depends on your constraints: extraction cost and latency tolerance, need for graph reasoning, multi-agent coordination requirements, and whether you're optimizing for response time or learning quality. The space is still young. The architectural divisions are clear, but production patterns are still emerging. Listen online: https://myweirdprompts.com/episode/stateful-memory-frameworks-architecture
DOI: 10.5281/zenodo.19558374
Language: English
Open Access status: green
OpenAlex ID: W7154166219
Cites:
Cited by:
Related to: 10