BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
Year: 2026
Type: conference-paper
Abstract: Pretrained biomedical vision-language models (VLMs) such as BioMedCLIP perform well on average but often degrade on challenging modalities where inter-class margins are small and acquisition-specific variations are pronounced, especially under few-shot supervision and when modality priors differ from pretraining corpora substantially.We propose BioVLM, a prompt-learning framework that improves cross-domain generalization without extensive backbone fine-tuning.BioVLM learns a diverse prompt bank and introduces dynamic prompt selection: for each input, it selects the most discriminative prompts via a lowentropy criterion on the predictive distribution, effectively coupling sparse few-shot evidence with rich LLM semantic priors.To strengthen this coupling, we distill high-confidence LLMderived attributes and enforce robust knowledge transfer through strong/weak augmentation consistency.At test time, BioVLM adapts by choosing modality-appropriate prompts, enabling transfer to unseen categories and domains, while keeping training lightweight and inference efficient.On 11 MedMNIST+ 2D datasets, BioVLM achieves new state of the art across three distinct generalization settings.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166796683