Work

Could Describe Anything Model Understand Remote Sensing Objects?

Year: 2025

Type: article

Abstract: Fine-grained image captioning for Remote Sensing Images (RSIs) remains an underexplored yet crucial task for multimodal scene understanding, especially in domains where localized semantic interpretation is essential. However, the heterogeneous nature of remote sensing data—spanning modalities such as visible light, infrared, and Synthetic Aperture Radar (SAR)—poses significant challenges for generalization, particularly under the lack of sentence-level annotations. Existing vision-language models, primarily trained on natural imagetext pairs, often fail to capture spatially grounded semantics in RSIs, leading to degraded performance when transferred to non-natural domains. In this paper, we present the first systematic evaluation of the Describe Anything Model (DAM) for localized captioning in remote sensing. To address the scarcity of aligned supervision, we construct a weakly supervised benchmark framework featuring three levels of region-of-interest prompting (full image, center point, bounding box), and harmonize test data across three modalities with a unified object category (ship). Caption quality is assessed through both linguistic diversity metrics and multimodal alignment indicators, including CLIP and RemoteCLIP scores. Experimental results reveal that DAM performs robustly on visible imagery with fine-grained prompts, but exhibits significant performance degradation in infrared and SAR domains, where modality-specific distortions hinder effective spatial grounding. Our benchmark exposes a critical bottleneck in current foundation models for cross-modality captioning and establishes a unified testbed for evaluating and improving multimodal language understanding in the remote sensing field

Source: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

DOI: 10.1109/jstars.2025.3616330

Authors: Ziyi Gao, Shuzhou Sun, Ming-Ming Cheng, Yunli Long, Yongxiang Liu, Li Liu

Institutions: National University of Defense Technology, Tsinghua University, Nankai University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,800

Funder: National Natural Science Foundation of China