Work

Model-dependent prompt effects challenge reliable deployment of medical LLMs in traditional Chinese medicine

Year: 2026

Type: article

Abstract: Although contemporary large language models (LLMs) have achieved satisfactory results in Western medical knowledge, their performance on traditional Chinese medicine (TCM) remains underexplored, and the determinants of performance variation are poorly understood. We evaluated 16 LLMs under seven prompt strategies on 475 questions from the 2025 Taiwan national TCM licensing examination, a public benchmark of TCM knowledge. Without prompt engineering, accuracy ranged from 35.4% to 85.9%, with approximately half passing. Among the evaluated model versions, adjusted accuracy differed substantially when models were grouped by family (for example, Claude-family models, 84% vs. GPT-4, 43%); these family-grouped comparisons were descriptive because only 16 individual models were evaluated. Paired prompt analyses showed modest average absolute effects, ranging from a 2.03-percentage-point decrease with implicit CoT to a 3.04-percentage-point increase with explicit CoT. However, prompt–model interactions produced substantial model-level divergence: prompt optimization reversed failing performance in one model (87.4% vs. 49.7%) while reducing accuracy in others. No single prompt strategy yielded consistent benefits across the evaluated models. Among five selected lower-performing models evaluated under explicit CoT prompting, misunderstanding of foundational TCM concepts was the most frequent error. These findings indicate that reliable LLM deployment in TCM requires empirical validation of both model choice and prompt strategy: model selection strongly shapes baseline capability, while prompt-model interactions can materially alter performance.

Source: npj Digital Medicine

DOI: 10.1038/s41746-026-03397-5

Authors: Liang-Wei Tseng, Yi-Chin Lu, Liang-Chi Tseng, Yu-Chun Chen, Hsing-Yu Chen