Year: 2023
Type: conference-paper
Abstract: Evaluation of dialogue systems requires assessing various aspects, among which appropriateness holds significance as a core element of communicative language competence.However, current evaluations heavily rely on human judgments, which are time-consuming, laborintensive, prone to biases, and lacking objectivity.In this paper, we introduce Dialogue Act Appropriateness (DAA), a novel method that utilizes the underlying patterns of dialogue act transitions to evaluate the appropriateness of chatbot responses.We learn transition patterns from human-human dialogue corpora, evaluating chatbot appropriateness by measuring the similarity of their transition patterns to those observed in human-human dialogues.To validate DAA, we annotate a test dataset by manually evaluating the appropriateness of dialogues from multiple chatbot systems.The experimental results demonstrate a strong correlation between our evaluation metric and human ratings, establishing the reliability of DAA as a measure of dialogue appropriateness. 1
Authors: Bao Chen, Yuanjie Wang, Zeming Liu, Yuhang Guo
Institutions: Beijing Institute of Technology, Beihang University
Language: English
Topic: Speech and dialogue systems
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Award: G2563629645
OpenAlex ID: W4389519971