Automated Chest X-ray Report Generation Remains Unsolved
Year: 2025
Type: conference-paper
Abstract: Accurate interpretation of chest radiograph images and generation of narrative reports is essential for patient care but places a heavy burden on radiologists and clinical experts. While AI models for automated report generation show promise, standardized evaluation frameworks remain limited. Here we present the ReXrank Challenge V1.0, a competition in the generation of chest radiograph reports utilizing ReXGradient, the largest test dataset consisting of 10,000 studies across 67 sites. The challenge attracted diverse participants from academic institutions, industry, and independent research teams, resulting in 8 new submissions alongside 16 state-of-the-art models previously benchmarked. Through comprehensive evaluation using multiple metrics, we analyzed model performance across various dimensions: differences between normal and abnormal studies, generalization capabilities across healthcare sites, and error rates in identifying clinical findings. This benchmark reveals that automated chest X-ray report generation remains fundamentally unsolved, with significant performance gaps between normal and abnormal studies, where even top-performing models achieve less than 45% error-free reporting on abnormal cases, and substantial variability across healthcare institutions, indicating that robust, clinically-ready systems require continued development before widespread deployment.
Authors: Xiaoman Zhang, Julian Nicolas Acosta, Xiaoli Yang, Subathra Adithan, Luyang Luo, Hong-Yu Zhou, Joshua Miller, Ouwen Huang, Zongwei Zhou, Ibrahim Ethem Hamamci, Shruthi Bannur, Kenza Bouzid, Xi Zhang, Zaiqiao Meng, Aaron Nicolson, Bevan Koopman, Inhyeok Baek, Hanbin Ko, Mercy Prasanna Ranjit, Shaury Srivastav, Sriram Gnana Sambanthan, Pranav Rajpurkar
Institutions: Harvard University, University of Glasgow, Duke University, Johns Hopkins University, University of Zurich, Microsoft (United States), Commonwealth Scientific and Industrial Research Organisation, Australian e-Health Research Centre, CSIRO Health and Biosecurity, Seoul National University, Microsoft Research (India), Indian Institute of Technology Madras
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: hybrid
OpenAlex ID: W4417312627