Work

인텐트 분류 모델의 신뢰도 임계값이 인간-에이전트 소통에 미치는 영향 연구

Year: 2022
Type: dissertation
Abstract: With the spread of smart speakers, task-oriented voice agents (voice apps) are also emerging. Task-oriented voice agents interact with the user through voice and provide specialized functions for specific domains. The domains of already released voice apps are various. The increase in task-oriented agents is inevitable in the future. \nCommunication breakdowns are exacerbated with task-oriented voice agents. Communication breakdowns are quite common in human-agent communication and have negative effects on the user. The fundamental causes are the invisible nature of voice interactions and incomplete artificial intelligence (AI). The task-oriented agent must accurately perform the requested task by the user in the specific domain. Therefore, the communication breakdowns with the task-oriented agent have more negative effects on the user. \nThe study on communication breakdowns from the perspective of NLU is needed. In AI-based agents, the NLU engine serves to understand the utterances of the user based on intent. Communication breakdowns are primarily caused by NLU (Natural Language Understanding) errors. However, existing researches focus on how to provide error prevention guidance and DM (Dialog Management). Therefore, to fill the gap between research and the cause of communication breakdowns, the approach from the perspective of NLU is needed. \nThis paper attempts to utilize the confidence threshold for the intent classification which classifies the users utterance into the intent. In voice interactions with AI agents, the confidence score is calculated when the users utterance is recognized. The system selects the intent with the highest confidence score. If the highest scoring intent has a confidence score greater than or equal to the confidence threshold, the agent provides the programmed response. In contrast, if the highest scoring intent has a confidence score lower than the confidence threshold, the agent returns the fallback response, such as Can you say that a different way? or I am not able to understand.. In other words, the confidence threshold is directly related to the agents performance and response propensity. \nThis paper aims to explore the impact of the confidence threshold on human-agent communication. The user identifies the agents smartness through conversing with the agent, and then adapts their utterances for the agent. In this process, the responses from the agent are important as evidence. In other words, the confidence threshold would make a difference in human-agent communication. To explore this, I conduct 1) a preliminary study for prototyping and identifying the distribution of the confidence scores, and 2) an experiment to explore the impact of the confidence threshold on human-agent communications. \nIn the preliminary study, I conducted a study to find user needs when using a voice agent for banking services and a study to identify the distribution of the confidence scores. In the user needs study, participants (n=10) are asked to use the banking agent that is run on the WOZ (Wizard of Oz) method. The utterance data was collected, and then open-coded to determine the main intents of the banking agent. Then, the second study is conducted with the prototype built on the intents. The utterance data was also collected from participants(n=10) who were asked to use the prototype. As a result, differences in response propensity between the agents were discovered, when the confidence thresholds are 0.3, 0.5, and 0.7. \nIn the experiment, I recruited participants (n=30) and ask them to use the prototype with the different confidence thresholds. The participants are divided into three groups considering the frequency of smart speaker use. Four sets of tasks consisting of In-Scope (IS) tasks and Out-of-Scope (OOS) tasks are assigned to the participants. When the task is completed, the participant fills out a survey on usability and intelligence of the agent. The designed independent variables are the confidence threshold (3, between-subject) x the frequency of smart speaker use (2, between-subject) x the set (2, within-subject). \nThrough the experiment, 988 utterance data and survey data were collected and then analyzed in terms of agents response, users utterance, and users perception. First of all, in terms of agents response, the confidence threshold had a significant effect on the communication success rate when exceeding a certain level. The communication success rate of the 0.7-agent was lower than the others (0.3=72%, 0.5=67%, 0.7=45%) in IS tasks, but higher than the others in OOS tasks (0.3=24%, 0.5=35%, 0.7=94%). The cause of communication breakdowns became uniform as the confidence threshold was higher. Comparing to the other agents, the 0.7-agent had more communication breakdowns caused by NLP (Natural Language Process) error (0.3=56%, 0.5=62%, 0.7=83%). Secondly, in terms of users utterance, communication with agents with low confidence thresholds was unnecessarily prolonged. In IS tasks, the number of users utterances was the lowest for the 0.5-agent and the highest for the 0.7-agent (0.3=32.8, 0.5=29.3, 0.7=40.1). However, there was not a significant effect between the 0.3-users and the 0.7-users. Considering the communication of the 0.3-agent was higher than the 0.7-agent, it suggests that the user had superfluous communication with the 0.3-agent. All the users employed the Rephrase the most to overcome the communication breakdowns, and the rate of employing the Repeat and the Quit depending on the confidence threshold. The 0.7-users employed the Quit more(0.3=16%, 0.5=23%, 0.7=27%), and the others employed the Repeat more(0.3=26%, 0.5=29%, 0.7=16%). Lastly, in terms of users perception, the confidence threshold and the frequency of smart speaker use did not have a significant effect. \nThis paper presents a novel approach using the confidence threshold to improve human-agent communication. It presents the possibility of the confidence threshold to improve human-agent communication and contributes as an exploratory study that opens the new area. In addition, based on the utterance data, it was analyzed from various points of view. This research can serve as a guide for researchers and practitioners who are interested in the confidence threshold. Lastly, this paper also suggests what points should be considered to utilize the confidence threshold in future research.
Author: 류명균
Language: Korean
Open Access status: green
OpenAlex ID: W7063100361
Cites: 52
Cited by:
Related to: 10