MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding
Year: 2026
Type: conference-paper
Abstract: The global transition to the ICD-11 taxonomy demands robust automated medical coding, yet comprehensive benchmarks to evaluate Large Language Models (LLMs) on this task remain absent.We introduce MAX-EVAL-11 1 as a foundational resource, the first large-scale benchmark for full-spectrum ICD-11 medical coding.MAX-EVAL-11 comprises 10,000 MIMIC-III discharge summaries with mapped, expert-validated subset (12%) ICD-11 annotations spanning 99.87% of the diagnostic taxonomy.To better reflect clinical utility, we propose a novel hierarchical evaluation framework that assigns partial credit based on ICD-11's 5-level structure, addressing the brittleness of traditional exact-match metrics.To illustrate benchmark utility, we evaluate LLMs on a stratified 500-patient sample, revealing significant performance gaps.The bestperforming model (Claude 4 Sonnet) achieves a weighted score of 0.433, outperforming both general-purpose peers and specialized medical models (MedCoder).Crucially, all models exhibit near-zero exact match rates (0-4.8%)and rely primarily on hierarchical credit, underscoring the extreme difficulty of precise ICD-11 code generation.Furthermore, the superiority of general-purpose LLMs over legacy ICD-10 medical models (with ICD-11 codelist) suggests that broad reasoning capabilities currently outweigh domain-specific training for complex taxonomy scaling.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7166804962