Work

QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions

Year: 2025

Type: conference-paper

Abstract: This paper explores a novel perspective to speech quality assessment by leveraging natural language descriptions, offering richer, more nuanced insights than traditional numerical scoring methods.Natural language feedback provides instructive recommendations and detailed evaluations, yet existing datasets lack the comprehensive annotations needed for this approach.To bridge this gap, we introduce QualiSpeech, a comprehensive lowlevel speech quality assessment dataset encompassing 11 key aspects and detailed natural language comments that include reasoning and contextual insights.Additionally, we propose the QualiSpeech Benchmark to evaluate the low-level speech understanding capabilities of auditory large language models (LLMs).Experimental results demonstrate that finetuned auditory LLMs can reliably generate detailed descriptions of noise and distortion, effectively identifying their types and temporal characteristics.The results further highlight the potential for incorporating reasoning to enhance the accuracy and reliability of quality assessments.The dataset can be found at https://huggingface.co/ datasets/tsinghua-ee/QualiSpeech. Speed:How would you assess the speaking speed in the audio?Use the following scale: 5. Very fast, 4. Mostly fast, 3. Moderate speed, 2. Fairly slow, 1. Very slow. Naturalness:How would you rate the naturalness of the audio?Choose a number from the following scale: 5. Very natural, 4. Mostly natural, 3. Somewhat natural, 2. Fairly unnatural, 1. Very unnatural.Noise: How would you rate the noise level in the audio?Use the following scale to describe its impact: 5.Not noIceable, 4. Slightly noIceable, 3. NoIceable but not intrusive, 2. Somewhat intrusive, 1. Very intrusive. Distor1on:How would you rate the level of distorIon in the audio?Use the following scale to describe its quality: 5.Not distorted, 4. Slightly distorted, 3. Somewhat distorted, 2. Fairly distorted, 1. Very distorted. Listening effort:How would you rate the intelligibility of the audio you just listened to?Use the following scale to describe your experience: 5. Complete relaxaIon possible; no effort required, 4. APenIon necessary; no appreciable effort required, 3. Moderate effort required, 2. Considerable effort required, 1.No meaning understood with any feasible effort. Con1nuity:How would you rate the conInuity of the audio?Use the following scale to describe its smoothness: 5. Very smooth, 4. Mostly smooth, 3. Somewhat smooth, 2. Fairly disjointed, 1. Very disjointed.Overall: How would you rate the overall quality of the audio?Use the following scale to describe your experience: 5. excellent, 4. good, 3. fair, 2. poor, 1. bad. Noise descrip1on:Please analyze the noise present in this audio clip.

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2025.acl-long.1150