Artificial Intelligence Supported Digital Counseling in Pregnancy: Evaluating Chatgpt S Performance Across 40 Scenarios
Year: 2025
Type: article
Abstract: Introduction: In recent years, language models supported by artificial intelligence have been increasingly utilized in healthcare counseling. This study aims to evaluate the reliability and clinical safety of ChatGPT-3.5 by assessing the medical accuracy, content completeness, and communication quality of its responses to pregnancy-related patient scenarios through expert review. Methods: Forty clinical scenarios reflecting common pregnancy concerns across all trimesters were developed. These were entered into ChatGPT-3.5, and its responses were independently evaluated by two obstetricians using an 8-item, 5-point Likert scale assessing medical accuracy, misinformation, missing information, urgency perception, referral appropriateness, home care suggestions, harmful recommendations, and language quality. A Global Quality Score (GQS) from 1 to 5 was also assigned. Internal consistency was analyzed using Cronbach’s alpha, and inter-rater agreement for GQS was evaluated via Cohen’s Kappa after categorization into low, moderate, and high-quality groups. Results: ChatGPT’s responses demonstrated high internal consistency across evaluation domains (Cronbach’s alpha = 0.872). Cohen’s Kappa for GQS agreement was 0.15, indicating slight agreement. No misinformation or harmful suggestions were identified. Most responses were medically appropriate, although some contained minor informational gaps. In 90% of the scenarios, evaluators placed responses in the same GQS category. Conclusions: ChatGPT-3.5 shows promise as a supportive digital tool for pregnancy-related counseling. However, it is not fully aligned with current clinical guidelines, which may limit its reliability in guideline-driven decision-making. Expert oversight remains essential, and further studies are needed to validate its performance in diverse clinical contexts.
Authors: Aybüke Tayarer, Kadir Kangal
Institution: Ahi Evran University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: bronze
OpenAlex ID: W7117472011