Year: 2025
Type: preprint
Abstract: Current benchmarks for AI clinician systems, often based on multiple-choice exams or manual rubrics, fail to capture the depth, robustness, and safety required for real-world clinical practice. To address this, we introduce the GAPS framework, a multidimensional paradigm for evaluating Grounding (cognitive depth), Adequacy (answer completeness), Perturbation (robustness), and Safety. Critically, we developed a fully automated, guideline-anchored pipeline to construct a GAPS-aligned benchmark end-to-end, overcoming the scalability and subjectivity limitations of prior work. Our pipeline assembles an evidence neighborhood, creates dual graph and tree representations, and automatically generates questions across G-levels. Rubrics are synthesized by a DeepResearch agent that mimics GRADE-consistent, PICO-driven evidence review in a ReAct loop. Scoring is performed by an ensemble of large language model (LLM) judges. Validation confirmed our automated questions are high-quality and align with clinician judgment (90% agreement, Cohen's Kappa 0.77). Evaluating state-of-the-art models on the benchmark revealed key failure modes: performance degrades sharply with increased reasoning depth (G-axis), models struggle with answer completeness (A-axis), and they are highly vulnerable to adversarial perturbations (P-axis) as well as certain safety issues (S-axis). This automated, clinically-grounded approach provides a reproducible and scalable method for rigorously evaluating AI clinician systems and guiding their development toward safer, more reliable clinical practice. The benchmark dataset GAPS-NSCLC-preview and evaluation code are publicly available at https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview and https://github.com/AQ-MedAI/MedicalAiBenchEval.
Source: arXiv (Cornell University)
Authors: Chen, Xiuyuan, Sun, Tao, Su, Dexin, Yu, Ailing, Liu, Junwei, Chen, Zhe, Jin, Gangzeng, Wang, Xin, Liu, Jingnan, Xiao, Hansong, Zhou, Hualei, Tao, Dongjie, Guo, Chunxiao, Yang, Minghui, Xia, Yuan, Zhao, Jing, Fan, Qianrui, Wang, Yanyun, Zhen, Shuai, Chen, Kezhong, Wang, Jun, Sun, Zewen, Zhao, Heng, Guan, Tian, Wang, Shaodong, Chang, Geyun, Deng, Jiaming, Chen, Hongchengcheng, Feng, Kexin, Li, Ruzhen, Geng, Jiayi, Zhao, Changtai, Wang, Jun, Lin, Guihu, Li, Peihao, Liu, Liqi, Wei, Peng, Wang, Jian, Gu, Jinjie, Wang, Ping, Yang, Fan
Institutions: Ant Group (China), Peking University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W4415277312