Work

Interactivity in Grounded Voice Assistants: Streaming Speech-to-Speech RAG and Ambiguity Resolution

Year: 2026
Type: dissertation
Abstract: Many frontline and deskless workers need hands-free access to their organisation’s internal knowledge while their hands and eyes are busy with a physical task. A spoken assistant can give them that, but two problems sit in the way. Building one that is fast enough for natural turn-taking and able to clarify vague questions is an engineering problem. Knowing whether that clarification really works, rather than only seeming to, is a measurement problem. This thesis takes on both. For the first, the system is a streaming speech-to-speech pipeline that looks up organisational knowledge on demand and asks a clarifying question when a request is missing detail. Because it starts speaking before its answer is finished, the time to first audio on a long reply drops from roughly five to ten seconds down to one to three. The sub-second timing that natural conversation expects cannot be reached once recognition, retrieval, generation, and synthesis are stacked together, so the goal is to cut the perceived delay as far as possible. The language model, not retrieval, is the slowest stage. For the second, two ways of deciding when to ask a clarifying question are compared: one based on the model’s own confidence, one on how much a question should narrow things down. The confidence-based one holds up when the evidence is realistic, getting the ambiguous questions right about nine times in ten, while the other falls apart once it works from real retrieved text instead of a clean answer key. Measuring this has a catch of its own: an automated stand-in for the user, if shown the answer, tends to leak it and flatter the results, so it is split from a silent judge that alone holds the answer. That removes the main leak, though on one dataset the answer can still show through the wording that defines each question. Either way, an accuracy figure means little without knowing the evidence the system had and how the user was simulated. A small exploratory study with ten participants gives a first look at the assistant in use: asking a clarifying question first helped most of them reach the answer they were after, at the cost of one extra turn. Because the query sets were tied to the two settings and the sample is small, these are early observations, not statistical claims. The two contributions match the two problems: a streaming voice system built and measured stage by stage, and a way of testing machine-simulated clarification that keeps the simulated user from being handed the answer.
Institution: Aalto University
Language: English
Open Access status: green
OpenAlex ID: W7203729179
Cites: 44
Cited by:
Related to: 10