Work

ChatGPT-4's Consistency, Specificity, and Inclusion of Behavior Change Techniques in Delivering Smoking Cessation Advice in Traditional Chinese: A Content Analysis

Year: 2025

Type: article

Abstract: INTRODUCTION: While ChatGPT has shown promise in health domains, its application in smoking cessation, particularly in non-English contexts, remains underexplored. This study assessed the consistency, specificity, and inclusion of behavior change techniques in ChatGPT-4's advice for smoking-related queries in Traditional Chinese across three phases. METHODS: ChatGPT-4 was accessed via Azure OpenAI Services (temperature: 0.7, Top P: 0.95). Phase I assessed consistency of 10 responses to identical smoking-related questions, measured using the Jaccard coefficient. Phase II evaluated response specificity using 12 smoking-related vignettes, tailored to age, readiness to quit, and nicotine dependence, through Jaccard distances and alignment to the 2008 U.S. Clinical Practice Guideline. In phase III, responses to 20 detailed smoking-related vignettes were analyzed for the inclusion of behavior change techniques and also analyzed qualitatively. Two independent coders performed analysis in each phase, and discrepancies were resolved by a third coder. RESULTS: Substantial agreement was observed between coders (Kappa = 0.63-1.00). ChatGPT-4 provided advice in Traditional Chinese with opening, bullet-point recommendations, and concluding. In phase I, the median Jaccard coefficient was 0.50, indicating moderate consistency. In phase II, specificity was higher for readiness to quit (0.58) and age (0.44) compared to nicotine dependence (0.40). Phase III identified 17 unique behavior change techniques, averaging 10 per response. Qualitative analysis found that ChatGPT-4's advice was somewhat tailored but lacked sufficient detail or contextual appropriateness. CONCLUSIONS: ChatGPT-4 demonstrated moderate consistency, specificity, and inclusion of evidence-based content in its Traditional Chinese cessation advice. Improvements are needed to increase adherence to cessation guidelines and enhance contextual relevance. IMPLICATIONS: This study is the first to evaluate ChatGPT's ability to deliver smoking cessation advice in Traditional Chinese, a non-dominant language for such large language model. It highlights the feasibility of using generative artificial intelligence for digital health interventions beyond English-speaking settings. The findings demonstrate ChatGPT's potential to support smoking cessation in Chinese-speaking populations and suggest broader applicability for culturally and linguistically underrepresented communities in global tobacco control efforts.

Source: Nicotine & Tobacco Research

DOI: 10.1093/ntr/ntaf267