Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text
Year: 2026
Type: preprint
Abstract: Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
Source: arXiv (Cornell University)
Authors: Kim, Hyeonyu, Kim, Hwayeon, Choi, Youngwon, Cho, Myeongkyun, Nguyen, Huu-Kim
Institution: Korea Advanced Institute of Science and Technology
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7204169452
Cites:
Cited by:
Related to: