Work

Certified large language model-based diagnostic decision support in rheumatology: the ALLIANCE multicentre randomised controlled trial

Year: 2026

Type: preprint

Study designs: Randomized Controlled Trial, Clinical Trial

Abstract: Objectives To evaluate whether access to a certified large language model (LLM)-based clinical decision support system improves physician diagnostic performance in rheumatology compared with conventional diagnostic resources alone. Methods In this multicentre, open-label, randomised controlled trial, 82 physicians from seven hospitals in two countries were randomised 1:1 to conventional diagnostic resources plus Prof. Valmed or conventional resources alone. Participants assessed three rheumatology vignettes before and after assistance. The primary outcome was top-1 diagnostic accuracy. Secondary outcomes included top-3 accuracy, diagnostic reasoning, confidence, case-processing time and perceived support quality. Results Top-1 accuracy increased from 22.2% to 33.3% in the intervention group and from 23.3% to 35.0% in the control group, with no between-group difference in improvement (adjusted OR 0.99, 95% CI 0.45 to 2.19; p=0.979). Differences in top-3 accuracy, diagnostic reasoning and confidence were also not significant. Assisted case-processing time was substantially shorter with LLM support (94 vs 206 s; adjusted mean difference −112 s, 95% CI −141 to −83; p<0.001). Information timeliness and perceived diagnostic support quality were rated significantly higher in the intervention group. Exploratory analyses showed persistent overconfidence and substantial AI over-reliance. Conclusions Certified LLM-based diagnostic support did not improve diagnostic accuracy compared with conventional resources, but substantially reduced case-processing time and improved perceived support quality. These findings suggest potential workflow benefits while highlighting overconfidence and over-reliance as important safety considerations. Trial registration number NCT07166692 WHAT IS ALREADY KNOWN ON THIS TOPIC Accurate and timely diagnosis in rheumatology is challenging because workforce shortages coincide with non-specific, complex and often rare presentations. Large language models are increasingly used for diagnostic decision support and have shown strong performance in benchmarking studies. Randomised evidence for physician-facing diagnostic decision support, particularly in rheumatology and for certified LLM-based systems, is lacking. WHAT THIS STUDY ADDS The ALLIANCE trial is the first randomised controlled trial to evaluate physician-facing diagnostic decision support in rheumatology and the first to assess a medically certified LLM-based clinical decision support system in any clinical specialty. A certified LLM-based clinical decision support system did not improve top-1 diagnostic accuracy compared with conventional resources alone, as assistance led to similar gains in both groups. Use of a certified LLM-based clinical decision support system was associated with approximately half the assisted case-processing time of conventional resources alone, as well as higher perceived support quality. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY LLM-based diagnostic decision support may be most useful for improving efficiency, support quality, and broadening differential diagnoses rather than increasing top-1 diagnostic accuracy. Safe clinical implementation will require careful attention to overconfidence, over-reliance, transparency, and trust. Further robust trials, ideally under real-world clinical conditions, are needed to define the role of clinical decision support in routine care.

Source: medRxiv

DOI: 10.64898/2026.08.29.26361715