Retrieval in the Voice Loop: Tool-Triggered RAG, Topic Preload and Caching Under a Latency Budget: Production Observations, a Design Pattern and an Evaluation Protocol
Year: 2026
Type: preprint
Abstract: Retrieval-augmented generation (RAG) lets a voice agent answer from an organization's own documents, but every retrieval step sits between the caller finishing a question and hearing the answer. On a production voice platform, an early retrieval path took 8.2 s for one knowledge question: 1.8 s to embed the query, 3.3 s for vector search and 3.1 s for a separate synthesis model call. A redesigned path, which retrieves only when the conversation model calls a tool, returns raw passages instead of synthesized answers, caches results and optionally preloads common topics, answers warm queries in 0.2 to 0.5 s. This paper documents that redesign as a design pattern for retrieval under a voice latency budget, identifies two remaining defects found in production (synchronous retrieval calls that block a shared event loop, and retrieval calls with no timeout), and specifies a controlled evaluation. The evaluation compares five retrieval configurations, with and without caching, on a public document-grounded dialogue corpus, measuring added latency per turn together with answer quality, and measures event-loop blocking under 1 to 20 concurrent sessions. Hypotheses and the analysis plan are stated before data collection; results will be reported in a follow-up version of this preprint. Companion to: Rehman, A. (2026). Architecture Patterns for Low-Latency Real-Time Voice Agents. Zenodo. https://doi.org/10.5281/zenodo.23216507
Author: Rehman, Altaf
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225915266
