Work

Medxchat: A Unified Multimodal Large Language Model Framework Towards CXRS Understanding and Generation

Year: 2025

Type: conference-paper

Abstract: Multimodal Large Language Models (MLLMs) have shown promise in general image tasks, but their application in medical imaging remains limited. This study explores MLLMs for Chest X-Rays (CXRs) through MedXChat, a unified framework enabling seamless interaction for tasks such as text report generation, visual question-answering (VQA), and Text-to-CXR generation. Our MLLM, which uses natural language as input, breaks task boundaries and simplifies medical professional training to the maximum extent by allowing diverse tasks within a single environment. To understand CXRs, we employ visual encoders (e.g.,ViT) and LLMs (e.g., mPLUG-Owl), which we refine using visual adapters and delta-tuning. For CXR generation, we introduce an innovative synthesis approach that leverages the instruction-following capabilities of the Stable Diffusion (SD) architecture to produce fine-grained medical images without additional parameters. Through comprehensive experiments, our model demonstrates exceptional cross-task adaptability, showcasing proficiency across all three tasks.

Source: IEEE International Symposium on Biomedical Imaging (ISBI)

DOI: 10.1109/isbi60581.2025.10981215

Authors: Ling Yang, Zhanyu Wang, Zhenghao Chen, Xinyu Liang, Luping Zhou

Institutions: The University of Sydney, University of Newcastle Australia, Guangzhou University of Chinese Medicine, First Affiliated Hospital of Guangzhou University of Chinese Medicine

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

SDG: Good health and well-being

Open Access status: green

OpenAlex ID: W4410295367

FWCI: 3.133

Cites: 16

Cited by: 9

Related to: 10