Work

Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study

Year: 2026

Type: article

Abstract: Objectives: To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus. Materials and Methods: We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy. Results: Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification. Discussion: Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise. Conclusion: Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.

Source: JAMIA Open

DOI: 10.1093/jamiaopen/ooag212