Concordance of GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 With the Formal EU-US Meniscus Rehabilitation 2024 Consensus: A Comparative Study at the Question and Statement Levels
Year: 2026
Type: preprint
Abstract: Background Large language models (LLMs) are increasingly used for clinical information retrieval in orthopaedics and sports medicine, but their value depends on fidelity to the relevant guideline or consensus. This study compared four LLMs with the formal 2024 ESSKA-AOSSM-AASPT EU-US Meniscus Rehabilitation Consensus in a realistic setting in which the consensus was named but its source text was not provided. Methods This consensus-referenced cross-sectional comparative study evaluated GPT-5.6, Gemini 3.6 Flash, DeepSeek-V4, and Doubao-2.1 using 37 clinical questions and 52 disaggregated recommendation statements. All models received identical instructions to answer strictly according to the consensus, without access to its PDF, full text, recommendation list, reference answers, or scoring criteria. Outcomes included question-level accuracy, strict statement-level concordance, at-least-partial coverage, error types, and potentially hazardous omissions. Paired differences were analyzed using Cochran's Q test and exact McNemar tests with Holm correction. Results Question-level accuracy differed among models (Cochran's Q = 9.09, p = 0.028): DeepSeek-V4 achieved 25/37 (67.6%), GPT-5.6 24/37 (64.9%), Gemini 3.6 Flash 23/37 (62.2%), and Doubao-2.1 16/37 (43.2%); no pairwise comparison remained significant after Holm correction. Strict statement-level concordance differed more markedly (Cochran's Q = 27.53, p < 0.001): GPT-5.6 achieved 25/52 (48.1%), Gemini 3.6 Flash 22/52 (42.3%), DeepSeek-V4 10/52 (19.2%), and Doubao-2.1 7/52 (13.5%). Errors primarily involved weight-bearing and range-of-motion restrictions, procedure-specific rehabilitation, return-to-sport criteria, outcome measures, and referral criteria. Conclusions LLM performance was heterogeneous when the consensus was named but not supplied. Question-level correctness did not ensure complete coverage of procedure-specific restrictions. LLMs may provide educational support, but individualized rehabilitation decisions require verification against the original consensus and clinical instructions.
Source: Research Square
Institution: University of Science and Technology of China
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7236470002
