Accuracy, Usefulness, and Impact Variability of ChatGPT-4o for COPD Medication Management: A Modified Delphi Study
Year: 2026
Type: article
Study design: Observational Study
Abstract: Background: Chronic obstructive pulmonary disease (COPD) management is complex and rapidly evolving. ChatGPT is a large language model (LLM) shown to generate treatment plans for chronic conditions, yet its accuracy, usefulness, and consistency for COPD remain poorly characterized. This study evaluated the accuracy, usefulness, and impact variability of simultaneous ChatGPT-4o responses to COPD medication management questions. Methods: Five COPD treatment questions were simultaneously entered into 3 separate computers using ChatGPT-4o during a single session, generating 15 total responses. Three residency-trained, board-certified clinical pharmacists rated each response across 3 domains—accuracy, usefulness, and impact variability—using a 3-point ordinal scale (0 to 2) via a 3-round modified Delphi process. Consensus was defined a priori as unanimous agreement among all 3 panelists. Results: Of 45 response-domain ratings, consensus was achieved in 40 (88.9%). Accuracy ranged from poor (0) to good (2), usefulness from somewhat (1) to very useful (2), and impact variability from low (0) to high (2). For one question on stable COPD pharmacotherapy, all 3 simultaneous responses cited a retired clinical practice guideline, resulting in poor accuracy ratings. For 2 questions—treating exacerbations and managing a complex case—one response per question was rated higher in usefulness than the others. Conclusion: Simultaneous ChatGPT-4o responses to identical COPD prompts differed materially in accuracy and potential clinical impact. LLM outputs should augment rather than replace clinician judgment and must be deployed with current guideline grounding and appropriate local oversight.
Institutions: University of Oklahoma, University of Georgia
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W7172430914