Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling
Year: 2025
Type: conference-paper
Abstract: Conversational assistants are increasingly popular across diverse real-world applications, highlighting the need for advanced multimodal speech modeling.Speech, as a natural mode of communication, encodes rich user-specific characteristics such as speaking rate and pitch, making it critical for effective interaction.Our work introduces DAMSEL, a data-centric customization approach for efficiently enhancing multimodal understanding in conversational speech modeling.Central to our contributions is a novel multi-task learning paradigm that involves designing auxiliary tasks to utilize a small amount of speech data.Our approach achieves state-of-the-art performance on the Spoken-SQuAD benchmark, using only 10% of the training data with open-weight models, establishing a robust and efficient framework for audio-centric conversational modeling.We also introduce ASK-QA, the first dataset for multi-turn spoken dialogue with ambiguous user requests and dynamic evaluation inputs. * Work done during an internship at Google.
Authors: Maximillian Chen, Ruoxi Sun, Sercan O Arik
Institutions: Google (United States), Columbia University
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888927