Work

Distilling an End-to-End Voice Assistant Without Instruction Training Data

Year: 2025

Type: conference-paper

Abstract: Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity.Recent efforts to address this with end-to-end Speech Large Language Models (speech-in, text-out) trained with supervised finetuning (SFT) have led to models "forgetting" capabilities from text-only LLMs.Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision.Importantly, this process can be performed without annotated responses.We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation.Furthermore, DiVA better matches user preferences, achieving a 72% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using >100x less training compute.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2025.acl-long.388

Authors: William Held, Yanzhe Zhang, Minzhi Li, Weiyan Shi, Michael J Ryan, Diyi Yang

Institutions: Northeastern University, Georgia Institute of Technology, National University of Singapore, Stanford University

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funders: National Science Foundation, Office of Naval Research

Award: CAREER: Socially-Aware Language Technologies To Support People in Supporting Others for Better Online Communities