Vision-Language and Clinical Data Fusion for Explainable AI based Medical Diagnosis and decision Support
Year: 2025
Type: article
Abstract: The fast-expanding multi-modality medical data volume leads to the development of the intelligent diagnostic system that combines medical image and clinical information. Because of the rapid development of multi-modality medical data, it is possible to develop intelligent diagnostic systems that combine the medical image and clinical information. This work puts forward a novel approach for a Vision-Language and Clinical Data Fusion framework to establish Explainable AI-based Medical Diagnosis and Clinical Decision Support. The presented study introduces a new model for Vision-Language and Clinical Data Fusion in the context of explainable AI-based medical diagnosis and clinical decision support. The proposed idea hinges on a visual representation extractor (vision encoder), a language representation extractor (clinical notes, EHR encoder) and a clinical feature representation extractor (structured patient information, clincial feature encoder). A cross-modal attention mechanism aligns the visual and textual representations while an adaptive fusion based on heterogeneous clinical features generates a unified representation for diagnosis. An explainable AI module is also used to provide attention maps, textual evidence for the clinical judgment and feature importance scores for enhancing the transparency of the predictions. An experimental assessment shows that the proposed multimodal framework demonstrates superior performance compared to the unimodal imaging and clinical-data frameworks in terms of accuracy (95.2%), precision (94.8%), recall (94.5%), F1-score (94.6%) and AUROC (97.1%). Ablation analysis verifies that the diagnostic performance and robustness significantly improve with vision-language alignment and clinical feature fusion. The key innovation is the fusion of multimodal representation learning and interpretable clinical evidence generation, which allows for more clear and trustworthy diagnosis with AI assistance. The framework is "scale-able" to support clinical decision-support applications.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7217366800