SpecEvaluator enables structured multi-expert quality assessment of speech dialogue systems with specialized large audio models
Year: 2026
Type: article
Abstract: We investigate automatic evaluation for spoken dialogue systems and propose SpecEvaluator, a structured multi-dimensional evaluation framework. Existing methods either rely on ASR transcription or produce a single holistic score, which hinders the preservation and discrimination of key aspects of speech dialogue such as semantic content, emotional expression, and speech quality. To address this, SpecEvaluator decomposes the evaluation task into three dimensions—content coherence, emotional appropriateness, and speech quality—each judged by a dimension-specific audio-language model. A zero-shot aggregation module then generates the final score and explanation. We further construct a multi-dimensional evaluation dataset with controllable quality gradients and generate dimension-level supervision signals using a teacher model. Experiments demonstrate that SpecEvaluator consistently outperforms strong baselines across ranking accuracy, correlation, and grade classification.
Source: Scientific Reports
Authors: Ruopan Wang, Hao Zhang, Dan Qu
Institution: PLA Information Engineering University
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
Award: G9001400853
OpenAlex ID: W7237168385
