CromSS: Cross-Modal Pretraining With Noisy Labels for Remote Sensing Image Segmentation
Year: 2025
Type: article
Abstract: We explore the potential of large-scale noisily labeled data to enhance feature learning by pretraining semantic segmentation models within a multimodal framework for geospatial applications. We propose a novel cross-modal sample selection (CromSS) method, a weakly supervised pretraining strategy designed to improve feature representations through cross-modal consistency and noise mitigation techniques. Unlike conventional pretraining approaches, CromSS exploits massive amounts of noisy and easy-to-come-by labels for improved feature learning beneficial to semantic segmentation tasks. We investigate middle and late fusion strategies to optimize the multimodal pretraining architecture design. We also introduce a cross-modal sample selection module to mitigate the adverse effects of label noise, which employs a cross-modal entangling strategy to refine the estimated confidence masks within each modality to guide the sampling process. Additionally, we introduce a spatial–temporal label smoothing technique to counteract overconfidence for enhanced robustness against noisy labels. To validate our approach, we assembled the multimodal dataset, NoLDO-S12, which consists of a large-scale noisy label subset from Google’s Dynamic World (DW) dataset for pretraining and two downstream subsets with high-quality labels from Google DW and OpenStreetMap (OSM) for transfer learning. Experimental results on two downstream tasks and the publicly available DFC2020 dataset demonstrate that when effectively utilized, the low-cost noisy labels can significantly enhance feature learning for segmentation tasks. The data, codes, and pretrained weights are freely available athttps://github.com/zhu-xlab/CromSS.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,645
Awards: EVOLUTION OF THE COPERNICUS LAND SERVICE PORTFOLIO INTEGRATING NOVEL EO DATA AND LATEST MACHINE LEARNING ALGORITHMS TO CONTINUOUSLY MONITOR THE STATUS, DYNAMICS AND BIOMASS OF THE LAND SURFACE, G2375103282, G2492830129, Künstliche Intelligenz in der Erdbeobachtung: Schlussfolgern, Unsicherheiten, Ethik und darüber hinaus, G91569014
OpenAlex ID: W4408564698
