Year: 2025
Type: conference-paper
Abstract: We introduce VoxRAG, a modular speech-tospeech retrieval-augmented generation system that bypasses transcription to retrieve semantically relevant audio segments directly from spoken queries.VoxRAG employs silence-aware segmentation, speaker diarization, CLAP audio embeddings, and FAISS retrieval using L2normalized cosine similarity.We construct a 50-query test set recorded as spoken input by a native English speaker.Retrieval quality was evaluated using LLM-as-a-judge annotations.For very relevant segments, cosine similarity achieved a Recall@10 of 0.34.For somewhat relevant segments, Recall@10 rose to 0.60 and nDCG@10 to 0.27, highlighting strong topical alignment.Answer quality was judged on a 0-2 scale across relevance, accuracy, completeness, and precision, with mean scores of 0.84, 0.58, 0.56, and 0.46 respectively.While precision and retrieval quality remain key limitations, VoxRAG shows that transcription-free speech-to-speech retrieval is feasible in RAG systems.
Authors: Zackary Rackauckas, Julia Hirschberg
Institution: Columbia University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412944250