Work

Chatbot rating scale (CHARS): Development and validation of a new tool for assessing the quality of health and social care chatbots

Year: 2026

Type: article

Abstract: Chatbots are increasingly deployed in healthcare, yet there is a dearth of standardized tools for evaluating their quality. The objective of this study was to develop and validate a novel, multidimensional instrument, the Chatbot Rating Scale (CHARS), for assessing the comprehensive quality of health and social care chatbots. This study employed a sequential multi-method design to develop and validate the Chatbot Rating Scale (CHARS). Phase 1 involved a rapid and comprehensive literature review and a modified Delphi process to identify key domains of chatbot quality and generate scale items. Phase 2 comprised psychometric evaluation, including pilot testing, construct validity assessment, reliability analysis, and factor structure determination. Items were refined through three Delphi rounds involving 24 experts, including clinicians, psychometrists, information technology specialists, graduate students, and end users. Following pilot testing of 10 chatbots, 979 participants, including health professionals, researchers, mental health experts, and end users, used CHARS (5-point Likert scale) to evaluate 36 health and mental health chatbots. Factorial validity was assessed using parallel analysis, exploratory factor analysis, and confirmatory factor analysis, while internal consistency was evaluated using Cronbach’s alpha in R (version 4.4.2). The modified Delphi study resulted in a refined scale comprising 25 items distributed across five initial dimensions. Empirical analysis supported a three-factor structure, which was: (1) Design, (2) Interactive User Experience (personalization, engagement, and usability), and (3) Functionality, with excellent internal consistencies (Cronbach’s alpha) of 0.90, 0.88, and 0.90, respectively. The factor loadings and interrater correlation coefficient for all 25 items of CHARS were high with standardized loadings ranging from 0.79 to 0.87 for Factor 2 (Design, ICC = 0.88), 0.77 to 0.87 for Factor 1 (Interactive User Experience, ICC = 0.98), and from 0.83 to 0.87 for Factor 3 (Functionality, ICC = 0.90), which were all statistically significant (p < .001). All 25 Delphi-derived items were retained and redistributed across the three factors, preserving comprehensive coverage of chatbot quality dimensions. CHARS is a novel, psychometrically robust, user-centred instrument that consolidates key experiential and functional attributes into a concise three-dimensional model. Its development highlights the convergence of user perceptions around integrated experiential qualities, offering a practical and theoretically grounded tool for evaluating chatbots. Future research should test its generalizability across cultural and technological contexts and examine predictive validity for user engagement.

Source: Scientific Reports

DOI: 10.1038/s41598-026-68018-3