Work

LoRA-Efficient Multi-Task Learning with Joint Semantic and Paralinguistic Distillation for Speech Emotion Recognition

Year: 2026

Type: article

Abstract: Speech Emotion Recognition (SER) is an important but challenging research topic in affective computing, aiming to identify the emotional state of a speaker from their speech. Emotional information in an utterance is carried by two complementary channels: the semantic content of the words (what is said) and the paralinguistic manner of delivery (how it is said). Self-supervised speech models such as WavLM capture acoustic patterns well, but they do not explicitly model lexical-semantic emotional cues. Multimodal alternatives, on the other hand, depend on automatic speech recognition (ASR) transcripts at inference time, increasing the deployment cost. To address these issues, we propose a LoRA-efficient multi-task learning framework built on a WavLM backbone, in which the semantic and paralinguistic representations of emotional speech are learned jointly inside a single speech encoder. The framework trains three objectives together: semantic distillation, paralinguistic self-reconstruction, and emotion classification, where the latter is treated as the primary task and the other two provide supporting supervision. Experiments on MSP-IMPROV and IEMOCAP validate the effectiveness of the proposed framework, and on IEMOCAP the model outperforms several existing SER methods, confirming the rationality of the design.

Source: Electronics

DOI: 10.3390/electronics15204741