MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs
Year: 2026
Type: conference-paper
Abstract: Large Language Models (LLMs) are increasingly applied to medicine, yet their adoption is limited by concerns over reliability and safety.Existing evaluations either test factual medical knowledge in isolation or assess patientlevel reasoning without verifying correctness, leaving a critical gap.We introduce MediEval, a benchmark that links MIMIC-IV electronic health records (EHRs) to a unified knowledge base built from UMLS and other biomedical vocabularies.MediEval generates diverse factual and counterfactual medical statements within real patient contexts, enabling systematic evaluation across a 4-quadrant framework that jointly considers knowledge grounding and contextual consistency.Using this framework, we identify critical failure modes, including hallucinated support and truth inversion, that current proprietary, open-source, and domain-specific LLMs frequently exhibit.To address these risks, we propose Counterfactual Risk-Aware Finetuning (CoRFu), a DPO-based method with an asymmetric penalty targeting unsafe confusions.CoRFu improves by +16.4 macro-F1 points over the base model and eliminates truth inversion errors, demonstrating both higher accuracy and substantially greater safety.
Authors: Zhan Qu, Michael Färber
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
Funder: Technische Universität Dresden
OpenAlex ID: W7166801843