MedCLIPSAM-CNNViT: A Text-Driven, Weakly Supervised Segmentation Framework for 3D Medical Images
Year: 2026
Type: conference-paper
Abstract: The original ViT encoder of general-purpose segmentation models such as SAM often loses high-frequency boundary information and produces single-scale, low-resolution token embeddings when applied to tiny lesions and ambiguous boundaries in medical images. To address these issues, we propose MedCLIPSAM-CNNViT, a weakly supervised medical image segmentation framework. By leveraging weak supervision, we enhance the model’s ability to learn from partially labeled data, reducing the reliance on fully annotated datasets. We design a novel HybridCNNViT backbone to replace SAM’s original encoder, combining the local inductive bias of CNNs with the global context of ViTs to produce multi-scale feature maps using a feature pyramid, thereby enhancing the representation of fine-grained anatomical structures. In addition, a multi-scale model fusion strategy is employed during Text-Driven prompt generation to improve robustness. During prompt generation, multi-layer feature fusion is used to improve the quality of saliency maps, and an adaptive pseudo-label thresholding scheme is proposed to dynamically accommodate varying grayscale distributions across modalities. Extensive experiments on four heterogeneous medical datasets including brain MRI, breast ultrasound, chest CT, and X-ray demonstrate that MedCLIPSAM-CNNViT achieves superior generalization and robustness, improving Mean Dice Score and Normalized Surface Distance by 2.74% and 1.42%, respectively, across datasets.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7204803208