Who Needs STT or TTS? The Waveform Remembers: Audio Memory for Voice Agents
Year: 2026
Type: article
Abstract: This paper proposes an audio memory architecture for a voice assistant whose persistent conversational history is managed outside the response model. The architecture retains source recordings, indexes them with audio embeddings, retrieves relevant episodes, and supplies original audio to a compatible answering model. It extends this foundation with storage of the assistant's own speech, anticipatory retrieval, background consolidation, contradiction review, and optional transfer of stable experience into a trainable local adapter. These extensions form a design proposal; the empirical contribution is a narrower end-to-end API proof of concept. On 9 November 2026, a prerecorded synthetic-audio experiment combined Gemini Embedding 2, a local SQLite store, Gemini audio-pair review, and xAI spoken responses with the Ara voice requested. All six fresh final cases retrieved the required recordings, produced the intended review decisions, and generated spoken outputs. A frozen lexical answer check passed five of six cases; a separate posthoc inspection of diagnostic transcripts found the expected answer content in all six. Mean time to first audio was 10.704 seconds, including a 7.205-second blocking review that did not influence the voice model's evidence. The experiment establishes feasibility of the recorded-audio retrieval and response path and supports the broader architecture proposed here.
Author: Marquez, Vincent
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225620152
FWCI:
Cites:
Cited by:
Related to: