Work

CLIP-Based Grid Features and Masking for Remote Sensing Image Captioning

Year: 2024

Type: article

Abstract: Remote sensing image (RSI) captioning is a vision-language multimodal task that aims to describe image content in natural language, facilitating accurate and convenient comprehension of RSIs. Existing methods primarily focus on extracting visual features using vision-task pretraining models, such as ResNet pretrained on the ImageNet, which may not be optimal for the vision-language task. In addition, there has been limited emphasis on text preprocessing, leading to missed opportunities to explore potential relationships among words within sentences. In this article, we propose a transformer-based model utilizing CLIP visual grid features and a random masking strategy for the RSI captioning task. To enhance RSI representations, we utilize the visual encoder of CLIP, a vision-language pretraining model, to directly extract visual grid features from RSIs. Subsequently, all training sentences undergo preprocessing via a random masking strategy to impart self-supervised text-learning capabilities to the model during the training stage. Extensive experiments conducted on the RSICD, UCM-Captions, and Sydney-Captions datasets demonstrate the superior performance of our method.

Source: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

DOI: 10.1109/jstars.2024.3510414

Authors: Qiaoling Lin, Shuang Wang, Xiutiao Ye, Ruixuan Wang, Rui Yang, Licheng Jiao

Institution: Xidian University

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,496

Funder: National Natural Science Foundation of China