Enhancing Dermatological Skin Lesion Classification with Multi-Modal Attention-Based Models and Explainability
Year: 2025
Type: conference-paper
Abstract: Accurate classification of skin lesions is critical for early detection of melanoma and other malignancies, particularly in resource-limited settings. This study presents a novel multi-modal machine learning framework that integrates dermoscopic images and structured clinical metadata to improve diagnostic performance. Leveraging the PAD-UFES-20 dataset, which includes over 2,000 smartphonecaptured lesion images and associated patient metadata, we benchmark a series of unimodal and multimodal models. Our results demonstrate that modality attention fusion (MAF) applied to a frozen SwinV2-Tiny vision transformer and metadata multi-layer perceptron (MLP), augmented with focal loss, yields a state-ofthe- art weighted F1-score of 0.84 and balanced accuracy of 0.802. Ablation studies and external validation on HAM10000 and ISIC 2019 datasets confirm the robustness of the proposed framework. We further introduce a Siamese architecture for contrastive embedding optimisation, which enhances class separability, particularly for underrepresented lesions. Explainability is achieved via SHAP for metadata and Grad-CAM for image features, enabling transparent, clinically aligned decision insights. This framework shows strong potential for deployment in teledermatology and mobile health diagnostics, particularly in underserved regions.
DOI: 10.21427/8pgb-2871
Author: O'Reilly, Conan
Institution: Technological University Dublin
Language: English
Subfield: Oncology
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7080014072
FWCI:
Cites:
Cited by:
Related to: