REG2024: a benchmark challenge and curated dataset for pathology report generation from giga- pixel whole-slide images of bladder cancer
Year: 2026
Type: preprint
Abstract: Background We aimed to construct a curated whole-slide image (WSI)-report dataset for bladder cancer and establish a clinically grounded benchmark for evaluating vision–language models through an open community challenge. Methods We curated a dataset of 902 deidentified bladder cancer WSIs paired with corresponding pathology reports. Seven teams submitted distinct modeling frameworks. Performance was assessed using a clinically weighted composite metric integrating word-overlap scores (ROUGE-L, BLEU-4), keyword matching, and embedding-based semantic similarity. Expert mean opinion scores (MOS) were obtained, and the correlation between the weighted-sum score and expert-assigned MOS was analyzed. Results Two modeling strategies emerged: end-to-end generative architectures and multi-step classification-then-generation pipelines. Among individual submissions, the top-ranked multi-step pipeline achieved the highest main-diagnosis accuracy (48/50), whereas end-to-end models showed greater sensitivity to rare pathological features. The weighted-sum metric correlated strongly with MOS within each team (Spearman’s ρ, 0.65–0.91) and, on reports with clinically important misclassifications, agreed with MOS more closely than any single metric. Conclusions Performance variability was influenced by architectural design and the integration of pretrained foundation models. Our proposed evaluation framework addressed key limitations of conventional natural language processing metrics in evaluating pathology reports. This challenge establishes a benchmark for clinical-grade pathology report generation by providing vision-language pathology datasets and introducing a clinically aligned evaluation framework, thereby advancing computational pathology.
Source: Research Square
Authors: Harim Oh, Eunsu Kim, Hyeseong Lee, Kyung A Kim, Sungman Hong, Sehwan Yoo, Yousun Kang, Hyo Sup Shim, Yumi Lee, Thanaporn Viriyasaranon, Wonkyeong Lee, Jang-Hwan Choi, Eunbeen Jo, Daehong Lee, Keonwoo Noh, Yeongjae Shin, Kyeongman Park, Joonbeom Park, Jong-Ho Kim, Hyun Yee Cho, Won-Ki Jeong, Sung Hak Lee, Sangjeong Ahn
Language: English
Topic: AI in cancer detection
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G4266867697, G7552388407, G7665247977
OpenAlex ID: W7235491203
