Work

PubMedCLIP: How Much Does CLIP Benefit Visual Question Answering in the Medical Domain?

Year: 2023

Type: conference-paper

Abstract: Contrastive Language-Image Pre-training (CLIP) has shown remarkable success in learning with cross-modal supervision from extensive amounts of image-text pairs collected online.Thus far, the effectiveness of CLIP has been investigated primarily in general-domain multimodal problems.In this work, we evaluate the effectiveness of CLIP for the task of Medical Visual Question Answering (MedVQA).We present PubMedCLIP, a fine-tuned version of CLIP for the medical domain based on PubMed articles.Our experiments conducted on two MedVQA benchmark datasets illustrate that PubMed-CLIP achieves superior results improving the overall accuracy up to 3% in comparison to the state-of-the-art Model-Agnostic Meta-Learning (MAML) networks pre-trained only on visual data.The PubMedCLIP model with different back-ends, the source code for pre-training them and reproducing our MedVQA pipeline is publicly available at https://github.com/sarahESL/PubMedCLIP.

Source: Findings of the Association for Computational Linguistics: EACL

DOI: 10.18653/v1/2023.findings-eacl.88

Authors: Sedigheh Eslami, Christoph Meinel, Gerard de Melo

Institutions: Hasso Plattner Institute, University of Potsdam

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Bundesministerium für Bildung und Forschung

Award: G7419989365