Applying Usability Testing to Conversational Systems in an Internet of Things Context: An Industry Experience Report
Year: 2026
Type: conference-paper
Abstract: The use of conversational systems has intensified in real-world applications, particularly in Internet of Things scenarios, where natural language interaction is considered a means of reducing system complexity. However, usability and user experience issues still affect the adoption and effectiveness of such systems. This paper presents an experience report on the conduct of usability tests on a conversational system applied in an Internet of Things context, aiming to analyze user experience and identify usability problems from the perspective of end users. The experience was structured using the Goal–Question–Metric framework, which supported the definition of analysis goals, guiding questions, and evaluation metrics. The usability tests involved representative tasks and the collection of quantitative and qualitative data, including task success rate, execution time, total questions per task, and users’ subjective perceptions. The results highlight recurring issues related to understanding system responses, feedback during interaction, and the adequacy of the conversational flow for the usage context. The findings provide exploratory insights into the design and evaluation of conversational systems, focusing on human and contextual aspects in human–computer interaction.
Institution: Universidade Federal do Amazonas
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Award: G2315157603
OpenAlex ID: W7204256643
