Year: 2026
Type: conference-paper
Abstract: Multi-modal Speech Language Models (SpeechLMs) are a recent advancement in natural language processing.These SpeechLMs are instruction-tuned and optimized for general tasks.Their usefulness for Automatic Speech Recognition (ASR), particularly in relatively low-resource scenarios, remains largely understudied.This work developed SpeechLM for ASR in Basque and Maltese and studied the impact of language-adapted Large Language Model (LLM) and speech encoder within the SpeechLM for ASR.Using supervised learning, we fine-tuned LLaMA-Omni, a SpeechLM, for ASR.We have conducted comprehensive hyperparameter tuning and experimented with language-adapted SpeechLM components to improve performance and evaluated our best models on in-distribution datasets for both languages and an out-of-distribution dataset for Basque.LLaMA-Omni achieved 8.09% WER in Basque and 25.65% WER for Maltese on average across multiple test splits.The in-distribution results show that SpeechLM outperforms a fine-tuned ASR system under specific constraints, whereas it underperforms the baseline model on out-of-distribution Basque, indicating weaker overall robustness.We also find that a language-adapted LLM within SpeechLM improves in out-of-distribution settings when compared to the off-the-shelf LLM within SpeechLM.
Institutions: University of the Basque Country, University of Malta
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: hybrid
Funder: Eusko Jaurlaritza
OpenAlex ID: W7166559167