Dual-View Vision Transformer with Cross-Modal Memory for Radiology Report Generation
Year: 2026
Type: conference-paper
Abstract: Automatically generating clinical narratives from chest radiographs is an active area of study designed to alleviate radiologist workload and standardize diagnostic reporting. Existing pipelines predominantly encode visual content through Convolutional Neural Networks (CNN), whose locality constraints inherently limit the modelling of anatomically dispersed pathological patterns. To address these limitations, we propose a dual-view framework that integrates a pre-trained Vision Transformer (ViT-B/16) with a Transformer encoder-decoder architecture augmented by a Cross-Modal Memory Network (CMN). Unlike prior memory-based approaches that rely on single-view CNN encoders, our framework simultaneously processes frontal and lateral chest X-ray projections through a shared ViT backbone, enabling richer cross-view global representations. The CMN incorporates a shared learnable memory matrix consulted by both the visual encoder and the textual decoder, effectively bridging the semantic gap between visual patch embeddings and specialized medical terminology. Extensive experiments on the MIMIC-CXR benchmark demonstrate that our framework achieves a B-4 of 0.135, a R-L of 0.302, and a CIDEr of 0.306, surpassing all comparable methods that share the same resource constraints, without the use of external knowledge repositories or foundation model decoders.
Institution: Digital Research Centre of Sfax
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7201836808