Cross-Modal Synthesis: Generating Semantic Segmentation Masks from Image Captions and Vice-Versa using Multi-Modal Transformers
Year: 2025
Type: article
Abstract: The integration of visual and linguistic modalities has transformed computer vision and natural language processing research. Cross-modal synthesis, which seeks to generate segmentation masks from text and captions from images, presents significant opportunities for understanding visual scenes semantically. In this work, we propose a unified transformer-based framework that performs bi-directional synthesis between image captions and semantic segmentation masks. The model, built on top of a multi-modal transformer encoder-decoder, learns shared latent representations enabling seamless translation between visual regions and linguistic tokens. Extensive experiments on COCO-Stuff and ADE20K datasets demonstrate that our method outperforms baseline models by 16% in mIoU for caption-to-mask synthesis and 14% BLEU improvement for mask-to-caption generation, establishing a new benchmark for multi-modal reasoning.
Authors: Subham Sahoo, Rajat Gupta, Chintan Tibrewala
Institution: Amity University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4416142334
FWCI:
Cites:
Cited by:
Related to: