Work

Comparative evaluation of large language models for guideline-concordant decision support in colorectal cancer multidisciplinary team care: A source-masked retrospective study

Year: 2026

Type: article

Abstract: Introduction Multidisciplinary teams (MDTs) are central to colorectal cancer (CRC) care, but real-world recommendations may vary because of information overload, heterogeneous documentation, and imperfect guideline implementation. Large language models (LLMs) have emerged as potential adjunctive clinical decision-support tools, yet their performance within multidisciplinary oncology workflows remains uncertain. Methods We conducted a source-masked, retrospective comparative study of treatment-naïve patients discussed at an MDT between January 1, 2022, and December 31, 2024. De-identified case narratives and key test results were reformatted into a standardized prompt and submitted to five LLMs using identical prompt templates. Each model was queried once per case in a fresh, independent chat session. Before evaluation, recommendations from the original MDT and the five LLMs were converted into a common source-neutral presentation template. Five senior oncologists, who were not provided with recommendation-source labels, independently rated each plan using a 5-point Likert scale for clinical appropriateness and guideline concordance, referencing CSCO and NCCN CRC guideline versions contemporaneous with the original MDT decision period. Results Ninety-one cases were included (41 colon and 50 rectal cancers). In the adjusted cumulative link mixed-effects model, all five LLMs had higher cumulative odds of receiving a higher guideline-concordance score than MDT recommendations (ORs, 9.68–25.69; all Holm-adjusted P <0.001). Descriptively, the largest score differences were observed in rectal cancer. LLM outputs more frequently included de-escalation strategies in stage I disease and neoadjuvant-treatment components in stage III rectal cancer. Inter-rater agreement varied across sources and was highest for MDT recommendations by ICC (2, k). Conclusions In this retrospective source-masked evaluation, LLM-generated CRC recommendations received higher reviewer-assigned guideline-concordance ratings than real-world MDT recommendations. These findings support prospective evaluation of LLMs as adjunctive tools for guideline-concordance auditing and documentation support rather than as substitutes for MDT judgment. Because clinical safety, patient outcomes, repeated-generation reproducibility, and successful source masking were not independently established, higher guideline-concordance ratings should not be interpreted as evidence of superior individualized care or clinical safety.

Source: Intelligent Oncology

DOI: 10.1016/j.intonc.2026.100090