Work

Scaling Up Biomedical Vision-Language Models: Fine-Tuning, Instruction Tuning, and Multi-Modal Learning

Year: 2025

Type: preprint

Abstract: To advance biomedical vison-language model capabilities through scaling up, fine-tuning, and instruction tuning, develop vision-language models with improved performance in handling long text, explore strategies to efficiently adopt vision language models for diverse multi-modal biomedical tasks, and examine the zero-shot learning performance. We developed two biomedical vision language models, BiomedGPT-Large and BiomedGPT-XLarge, based on an encoder-decoder-based transformer architecture. We fine-tuned the two models on 23 benchmark datasets from 6 multi-modal biomedical tasks including one image-only task (image classification), three language-only tasks (text understanding, text summarization and question answering), and two vision-language tasks (visual question answering and image captioning). We compared the developed scaled models with our previous BiomedGPT-Base model and existing prestigious models reported in the literature. We instruction-tuned the two models using a large-scale multi-modal biomedical instruction-tuning dataset and assessed the zero-shot learning performance and alignment accuracy.

Source: arXiv (Cornell University)

Authors: Peng, Cheng, Zhang, Kai, Lyu, Mengxian, Liu, Hongfang, Sun, Lichao, Wu, Yonghui

Institutions: Lehigh University, The University of Texas Health Science Center at Houston

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Patient-Centered Outcomes Research Institute, Nvidia, Advanced Research Projects Agency, Clinical and Translational Science Institute, University of Florida, National Institute on Aging, National Heart, Lung, and Blood Institute, National Institute on Drug Abuse, National Cancer Institute, National Institute of Allergy and Infectious Diseases, U.S. National Library of Medicine

Awards: G1254130036, Advancing Drug Repositioning for Alzheimer’s Disease using Real-world Data, G4156463097, G4691473461, G5717028053, Semi-structured Information Retrieval in Clinical Text for Cohort Identification, Developing and Evaluating a Machine-Learning Opioid Prediction & Risk-Stratification E-Platform (DEMONSTRATE)

OpenAlex ID: W6979280559

Cites:

Cited by:

Related to: