Year: 2025
Type: article
Abstract: Early and accurate detection of breast cancer is critical for improving patient outcomes and reducing mortality. In this paper, we propose a multimodal deep learning framework that integrates high-resolution mammographic image analysis using Vision Transformers (ViTs) with clinical text interpretation through BERT-based language models. By combining visual and textual information via an early fusion strategy, our approach captures complementary diagnostic cues to enhance prediction accuracy. We evaluate our model on two publicly available datasets-CBISDDSM and MIMIC-CXR-and demonstrate that the multimodal system significantly outperforms unimodal baselines. Our best-performing model achieves an accuracy of 91.4 and an AUROC of 0.94, surpassing both ViT-only and BERT-only models. Additional experiments and ablation studies confirm the effectiveness of the fusion strategy and the contribution of each modality. These findings highlight the potential of multimodal transformer-based learning to support radiologists in early breast cancer diagnosis through more holistic and robust decision-making.
Authors: Maryam Alaei, Mohammad Zare, Mehdi Hazrati, Amir Chekini
Language: English
Topic: AI in cancer detection
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W4410600686