Work

Transferable image synthesis for remote sensing semantic segmentation via joint reference-semantic fusion

Year: 2025

Type: article

Abstract: With the advancement of diffusion model-based generative methods, synthesizing pixel-level training datasets has emerged as a promising approach to mitigate the scarcity of annotated data in semantic segmentation tasks. However, a noticeable gap persists between data synthesis and semantic segmentation tasks in the remote sensing (RS) domain. Due to the diversity of RS data across sensors and spatial scales, relying solely on limited annotated data and pre-trained generative foundation models to synthesize training data brings minor improvements in RS semantic segmentation tasks. Therefore, it becomes crucial to incorporate large volumes of unlabeled external data into downstream tasks to enable more transferable image synthesis. Unlike training-free approaches that introduce reference (Ref) images primarily for shallow feature transfer, we propose a joint learning strategy that integrates Ref images, semantic masks, and text prompts during training. This facilitates multi-modal interaction and allows the model to capture deeper features such as content. To achieve effective multi-modal information fusion, the proposed Transferable Image Synthesis method (TISynth) avoids directly using real Ref images during training. Instead, it generates Ref images from augmented input images, and facilitates interaction between Ref images and semantic information through text prompts and an all-in-attention module. Experimental results demonstrate that our synthetic images improve the OA/mIoU/mAcc by 1.52%/2.32%/3.04% on FUSU and 1.33%/1.06%/3.04% on GID compared to the baseline trained only on the original data. In addition, compared to state-of-the-art segmentation training data synthesis methods, our method achieves the best results on the two RS semantic segmentation datasets.

Source: Information Fusion

DOI: 10.1016/j.inffus.2025.103839