Year: 2026
Type: article
Abstract: Objective: Large language model (LLM) chatbots show promise for pre-visit planning, but their English-centric development raises equity concerns for multilingual deployment. We evaluated the multilingual capabilities of the Primary Care Pre-visit Chatbot (PCP-Bot), a GPT-4o-based pre-visit chatbot, across English, Mandarin Chinese, Spanish, and Hindi. Materials and Methods: In this mixed-methods study, 31 bilingual participants (11 Mandarin, 10 Spanish, and 10 Hindi) role-played as patients across 5 synthetic primary care cases, each first in English, then in their second language. Participants completed validated instruments (Bot Usability Scale [BUS-11], NASA Task Load Index [NASA-TLX], Human-Computer Trust Scale [HCTS], and Technology Acceptance Model-Perceived Ease of Use [TAM-PEU]), an adapted summary quality rubric (fluency, adequacy, meaning, and severity), and provided open-ended feedback. Results: = .087). Trust, workload, and ease-of-use remained consistent across languages. Participants described PCP-Bot as natural and accurate overall but reported repetition, transcription errors, and more frequent breakdowns in non-English interactions. Discussion: Multilingual capability in pre-visit workflows appears feasible for LLMs under controlled conditions in the 4 high-resource languages evaluated, with high absolute performance in each. Disparities were language-dependent: conversational performance declined most for Spanish and Hindi, while summary quality showed a small, uniform English advantage, suggesting addressable language-specific weaknesses rather than a fixed limitation of English-developed systems. Conclusion: Large language models can support multilingual pre-visit interactions, but performance varies by language. Per-language validation and further testing are needed prior to clinical rollout.
Source: JAMIA Open
Institution: University of Pennsylvania
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: gold
Estimated APC paid: $3,762
OpenAlex ID: W7223392010