Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) are increasingly applied in clinical decision support, yet current evaluations rarely reveal whether their outputs reflect genuine medical reasoning or superficial correlations.We introduce DeVisE (Demographics and Vital signs Evaluation), a behavioral testing framework that probes finegrained clinical understanding through controlled counterfactuals.Using intensive care unit (ICU) discharge notes from MIMIC-IV, we construct both raw (real-world) and templatebased (synthetic) variants with single-variable perturbations in demographic (age, gender, ethnicity) and vital sign attributes.We evaluate eight LLMs, spanning general-purpose and medical variants, under zero-shot setting.Model behavior is analyzed through (1) inputlevel sensitivity, capturing how counterfactuals alter perplexity, and (2) downstream reasoning, measuring their effect on predicted ICU lengthof-stay and mortality.Overall, our results show that standard task metrics obscure clinically relevant differences in model behavior, with models differing substantially in how consistently and proportionally they adjust predictions to counterfactual perturbations. 1 How do counterfactuals change the probability distribution of downstream tasks?How do counterfactuals change how likely the patient is?Qwen-2.5-Instruct 72B GPT-OSS 120B GPT-4.1-mini?
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140101800