Work

Advancing user satisfaction estimation in task-oriented dialogue system

Type: dissertation
Abstract: The task of User Satisfaction Estimation (USE) is crucial in evaluating the performance of conversational systems. It refers to the process of assessing the degree to which a user is content or pleased with their interaction with the system. USE aims to quantify the user's overall experience, taking into account various factors such as the relevance of responses, the system's ability to understand and address user needs, and the efficiency of the interaction. This thesis begins with a comprehensive investigation into human-annotated datasets used for USE in conversational systems, including MWOZ and SGD. It examines the inherent weaknesses and limitations that impact their effectiveness in training and evaluating USE models. The weaknesses and limitations include the subjective nature of human annotations, the challenges posed by long-term dialogues, and the resource-intensive process of creating high-quality labeled data. Through this examination, the thesis identifies areas where improvements are necessary to advance the field of USE in task-oriented dialogue systems. To address these limitations, this thesis proposes an innovative approach. Specifically, it involves leveraging Large Language Models (LLMs), to replace human labor in annotating datasets for User Satisfaction Estimation. It aims to mitigate the subjectivity and resource constraints associated with human annotation while maintaining high-quality labels. Following the creation of this LLM-annotated dataset, the thesis conducts a rigorous evaluation of state-of-the-art baseline models. This evaluation serves to benchmark the performance of existing USE models for our newly created dataset. After that, the thesis explores the potential of LLM as a powerful data augmentation tool for USE. It aims to address the limitation of data scarcity and imbalance in data samples in USE research through the generative capabilities of the LLMs. Specifically, USE datasets are commonly composed of dialogues with Satisfied, Neutral, and Dissatisfied labels, where dissatisfied labels often contain fewer samples compared to Neutral and Satisfied labels. Therefore, this approach involves using LLMs to generate additional dialogues of corresponding satisfaction labels based on the training samples. It effectively increases the size of training data, especially in labels with scarce data samples. The augmented dataset will lead to more robust USE models. Additionally, the thesis conducts evaluations on state-of-the-art baseline models with the augmented datasets. This evaluation assesses the impact of data augmentation on model performance and compares the effectiveness of generated data against original USE datasets. The evaluation aims to provide valuable insights into the viability and effectiveness of using LLMs for data augmentation in the task of USE.
Language: English
Open Access status: green
OpenAlex ID: W7216627089
Cites: 43
Cited by:
Related to: 10