Year: 2026
Type: conference-paper
Abstract: Currently, large language models (LLMs) predominantly focus on the text modality.To enable more natural human-AI interaction, speech LLMs are emerging, but building effective endto-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages.In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for crosslingual speech LLMs based on discrete speech tokens.We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training.By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency.CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability.Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities.
Authors: Yan Zhou, Qingkai Fang, Yun Hong, Yang Feng
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166799440