Use of a Medical Communication Framework to Assess Quality of Automated Replies to Primary Care Portal Patient Messages: A Content Analysis. (Preprint)
Year: 2025
Type: preprint
Abstract: BACKGROUND Generative artificial intelligence (GenAI) tools are increasingly available to assist clinicians in responding to patient messages; however, their suitability as a tool for medical communication in primary care has not been systematically assessed. OBJECTIVE To assess current strengths and limitations of GenAI for primary care messages according to a medical communication framework. METHODS This was a descriptive quality improvement study of 201 GenAI replies to real patient messages, submitted to primary care physicians through the electronic portal at a large midwestern academic medical center. Two PCP reviewers applied a medical communication framework to develop a codebook defining strengths and limitations across five communication domains. The reviewers then assessed the presence of communication strengths and limitations for each GenAI draft, according to the codebook. All discrepancies between reviewers were reconciled via discussion, and reconciled strengths and limitations were tallied. We report the frequency of observed strengths and limitations in communication domains. RESULTS Across all messages (n=201), 26.4% (53 of 201) had strengths only and no limitations, while 27.4% (55 of 201) only had limitations and no strengths. The remaining 46.3% (93 of 201) had a mix of strengths and limitations. Strengths were more common than limitations in the domains of “Rapport Building” (43.3% [87 of 201] vs. 17.4% [35 of 201]) and “Enabling Next Steps” (36.3% [73 of 201] vs. 19.4% [39 of 201]). Limitations were more common in the remaining domains of “Information Delivery” (44.3% [89 of 201] vs 21.4% [43 of 201]), “Information Gathering” (29.9% [60 of 201] vs 21.4% [43 of 201]), and “Responding to Emotion” (8.5% [17 of 201] vs 4.5% [9 of 201]). CONCLUSIONS GenAI drafts may often contain usable portions such as expressions of respect or outlining common next steps in response to primary care patient messages. However, those strengths may be tempered by limitations in other important communication domains requiring clinician judgment, including gathering and delivering appropriate information and responding to emotion. Careful monitoring is needed to ensure that GenAI drafts do not negatively impact patient-physician communication.
Authors: Natalie S. Lee, Nathan Richards, Jodi Grandominico, Robert M. Cronin, Amanda K Hendricks, Ravi S. Tripathi, Daniel E. Jonas
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W4407061558