M3: multimodal artificial intelligence for medical report generation and visual question answering from 3D abdominal CT scans
Year: 2025
Type: article
Abstract: Objectives: Medical imaging is indispensable for diagnosis, with abdominal imaging playing a pivotal role in generating medical reports and informing clinical decision-making. Recent works in artificial intelligence (AI), particularly in multimodal approaches such as vision-language models, have demonstrated significant potential to enhance medical image analysis by seamlessly integrating visual and textual data. While 2D imaging has been the main focus of many studies, the enhanced spatial detail and volumetric consistency offered by 3D images, such as CT scans, remain relatively underexplored. This gap underscores the need for innovative approaches to unlock the potential of 3D imaging in clinical workflows. Methods: In this study, we utilized a multimodal AI pipeline, Phi3-V, to address 2 key challenges in abdominal imaging: generating clinically coherent medical reports from 3D CT images and performing visual question answering based on these images. Results: Our optimized model attained an average GREEN score of 0.409 for medical report generation and an accuracy of 79% for multiple-choice visual question answering on the validation cases. Conclusions: These findings demonstrate the potential of multimodal AI in advancing the analysis of 3D medical imaging, paving the way for more robust and efficient applications in healthcare. Advances in knowledge: This study advances the use of multimodal AI for 3D CT imaging, achieving improvements in medical report generation and visual question answering.
Source: BJR|Artificial Intelligence
Authors: Abdullah Hosseini, Ahmed Ibrahim, Ahmed Serag
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,492
Funder: Weill Cornell Medicine - Qatar
OpenAlex ID: W4412865275