CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging
Year: 2025
Type: conference-paper
Abstract: Recent advances in multimodal large language models (MLLMs) has enabled unified processing of visual and textual inputs, with promising implications for general-purpose medical AI. However, their ability to generalize compositionally across unseen combinations of imaging modality, anatomy, and task type remains underexplored. We introduce CrossMed, a benchmark designed to evaluate compositional generalization (CG) in medical MLLMs using a structured Modality-Anatomy-Task (MAT) schema. CrossMed reformulates four public datasets, CheXpert (X-ray classification), SIIM-ACR (X-ray segmentation), BraTS 2020 (MRI classification and segmentation), and MosMedData (CT classification) into a unified visual question answering (VQA) format, resulting in 20,200 multi-choice QA instances. We evaluate two open-source MLLMs, LLaVA-Vicuna-7B and Qwen2-VL-7B, on both Related and Unrelated MAT splits, as well as a zero-overlap setting where test triplets share no Modality, Anatomy, or Task with training data. Models trained on Related splits achieve 83.2% classification accuracy and 0.75 segmentation cIoU, while performance drops significantly under Unrelated and zero-overlap conditions, validating the benchmark’s difficulty. Furthermore, we show cross-task transfer where segmentation performance improves by +7% cIoU even when trained using classification-only data. Traditional models (ResNet-50, U-Net) benefit modestly, confirming MAT’s broad utility, while MLLMs uniquely excel at CG. CrossMed provides a rigorous testbed for evaluating zero-shot, cross-task, and modality-agnostic generalization in medical vision-language models.
Authors: Pooja Singh, Siddhant Ujjain, Tapan Kumar Gandhi, Sandeep Kumar
Institution: Indian Institute of Technology Delhi
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7110140238