A cross-sectional benchmark of seven large language models for National Comprehensive Cancer Network guideline adherence in non-small cell lung cancer management
Year: 2026
Type: article
Abstract: Background: While large language models (LLMs) show promise in general medicine, existing benchmarks lack the disease-specific rigor needed to assess guideline adherence. Non-small cell lung cancer (NSCLC) care requires alignment with rapidly evolving National Comprehensive Cancer Network (NCCN) Guidelines. We developed an NCCN-grounded benchmark to assess guideline concordance of seven selected LLMs in NSCLC management. Methods: We developed an NCCN (Version 4.2025)-grounded NSCLC benchmark. The final analysis set contained 793 archived question-and-answer (Q&A) items. Each item included weighted positive criteria and safety-critical negative criteria. Seven specified LLMs were evaluated once per item, without a system prompt, retrieval augmentation, tools, or internet access. Automated rubric-based scores were used for descriptive comparison. Results: Across seven specified LLMs on 793 NCCN Guidelines-based queries, Gemini-3-Pro achieved the highest clean normalized score (0.739), while DeepSeek-V3.2 was the highest-scoring open-source model (0.695). Scores were generally higher for foundational tasks and lower for clinical-trial interpretation and therapeutic modality selection, with interpretation of clinical trial results showing the lowest task-level performance. Gemini-3-Pro and DeepSeek-V3.2 had the lowest omission rates (28.36% and 33.24%, respectively). Gemini-3-Pro had a 0% inclusion rate of rubric-defined contraindicated negative points in this benchmark. Automated scores are benchmark-specific and do not establish clinical safety or independently validated clinical quality. Conclusions: This NCCN-grounded benchmark compares seven specified LLMs using a fixed question set. Although leading models performed better on several foundational tasks, the results reflect guideline-rubric concordance rather than clinical correctness and do not show readiness for independent clinical decision-making.
Authors: Zewen Sun, Tian Guan, Jiayi Geng, Junwei Liu, Dexin Su, Tao Sun, Gangzeng Jin, Heng Zhao, Shaodong Wang, Geyun Chang, Guihu Lin, Changtai Zhao, Kexin Feng, Hongchengcheng Chen, Jiaming Deng, Ruzhen Li, Jun Wang, Peihao Li, Liqi Liu, Yun Li, Fan Yang, Jun Wang, Xin Wang, Ailing Yu, Yuan Xia, Dongjie Tao, Hualei Zhou, Chunxiao Guo, Peng Wei, Jian Wang, Jinjie Gu, Yanyun Wang, Jing Zhao, Shuai Zhen, Kezhong Chen, Ping Wang, Xiuyuan Chen
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: diamond
OpenAlex ID: W7223923215