Performance evaluation of large language models in real-world perinatal medication consultations: a cross-sectional study
Year: 2026
Type: article
Abstract: INTRODUCTION: Perinatal medication consultation is a core clinical pharmacy service that involves a complex benefit-risk assessment for both maternal and fetal safety. Large language models (LLMs) have emerged as potential tools to improve access to medication information, yet their performance and safety in real-world, pharmacist-led perinatal consultation settings, particularly in non-English contexts, remain insufficiently evaluated. AIM: To evaluate and compare the performance of multiple advanced large language models in addressing real-world Chinese perinatal medication consultation queries and to assess their potential role as supervised adjunctive tools within clinical pharmacy services. METHOD: This cross-sectional study evaluated seven LLMs using real-world clinical data from pharmacist-led medication consultations at the Pharmacy Clinic of the Beijing Obstetrics and Gynecology Hospital, Capital Medical University. A standardized test set of 64 perinatal medication consultation questions was developed from 15,280 electronic consultation records collected between April 2014 and April 2024. The evaluated models included international (GPT-5.1, Grok 3, Gemini 3.0) and domestic (DeepSeek, Wenxin Yiyan, Kimi K2, Tongyi Qianwen) models. Senior clinical pharmacologists independently assessed responses across four dimensions-relevance, accuracy, usefulness, and empathy-using a 10-point Likert scale. Results are reported primarily as median (IQR), with mean ± SD additionally provided as a secondary descriptor to facilitate comparison with prior literature. RESULTS: = 0.41, large effect). GPT-5.1 achieved the highest median total score [9.3 (IQR: 8.8-9.6); mean ± SD: 9.1 ± 0.8], outperforming all other models (all Bonferroni-corrected p < 0.01; all r > 0.50, large effect sizes), followed by Kimi K2 [8.5 (IQR: 7.9-9.1); mean ± SD: 8.4 ± 1.2] and DeepSeek [8.3 (IQR: 7.6-8.9); mean ± SD: 8.2 ± 1.1]. Tongyi Qianwen demonstrated the lowest overall performance [6.7 (IQR: 5.9-7.4); mean ± SD: 6.8 ± 1.3]. Accuracy was the primary determinant of performance differences. Performance gaps were more pronounced in complex clinical scenarios involving comorbidities or benefit-risk trade-offs, whereas domestic models demonstrated relative advantages in consultations involving traditional Chinese medicine. CONCLUSION: LLMs have demonstrated variable performance in response to perinatal medication consultation queries. While high-performing models show potential to support pharmacist-led perinatal medication consultations by improving access to information, their current performance supports use only as supervised, adjunctive decision-support tools rather than independent sources of medication counseling, with human oversight essential prior to broader integration.
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Estimated APC paid: $4,190
Award: G2389466084
OpenAlex ID: W7155854067