H2-MRG: Hierarchical Heterogeneous Graph Based Medical Report Generation with Fine-Grained Disease-Specific Modeling
Year: 2026
Type: conference-paper
Abstract: Vision-language model (VLM)-based medical report generation (MRG) has shown great potential for automating radiology workflows.However, existing VLMs often use coarse fixedsize patches that ignore anatomical boundaries, weakening anatomically grounded visual evidence and structured disease reasoning that are crucial for accurate report generation.In addition, CLIP-style contrastive pretraining treats all unpaired image-report pairs as negatives, introducing false negatives when different patients share similar diseases or findings.To address these issues, we propose H 2 -MRG, a hierarchical heterogeneous graph-based MRG framework.Specifically, fine-grained image patches containing complete anatomical structures are extracted.Based on the patches, a hierarchical heterogeneous graph attention network (H 2 -GAT) module is presented, where anatomy and disease nodes are connected and aggregate information through anatomy-anatomy (spatial), anatomy-disease (diagnostic), and disease-disease (co-occurrence) relations, enabling anatomically grounded disease reasoning.In addition, an anatomy-level sampling strategy for patch-sentence-level alignment is used to address the false negative problem.Experiments on MIMIC-CXR and IU-Xray datasets demonstrate that our framework outperforms state-of-the-art methods in both natural language generation and clinical efficacy metrics.
Authors: Libiao Chen, Dong Nie, Zhenyu Tang
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224408548