Work

A Cross-Domain Tool-Augmented Vision–Language Framework for Remote Sensing Image Understanding

Year: 2026

Type: article

Abstract: Vision–language models (VLMs) hold considerable potential for interpreting large-scale remote sensing (RS) archives, which are critical for applications such as environmental monitoring, disaster response, and urban planning. However, general-purpose VLMs primarily target optical imagery and often underperform on RS tasks, while existing RS-specific VLMs still struggle with fine-grained understanding. To address these limitations, we propose GeoPilot, a tool-augmented multimodal assistant tailored for RS scenarios. GeoPilot interprets user instructions, autonomously determines whether to invoke external tools, and synthesizes their outputs to generate precise responses. A key capability of our approach is its ability to process both optical and Synthetic Aperture Radar (SAR) imagery, supporting representative tasks such as visual grounding, object detection, segmentation, and cross-domain reasoning. To support this setting, we construct a novel large-scale RS instruction dataset that jointly supports optical and SAR imagery together with explicit tool use reasoning traces, addressing the critical challenge of task-specific data scarcity. We also introduce GeoPilotBench, a benchmark for cross-domain, multi-task dialogue and tool-aware evaluation in RS, and use it to assess GeoPilot across representative tasks. Experimental results show that GeoPilot achieves strong task planning accuracy (92.6% overall planning accuracy) and competitive performance on VQA, SAR understanding, and referring object detection. End-to-end evaluation further confirms that GeoPilot’s learned tool policy introduces only limited overhead compared to standalone tool execution, demonstrating its practical value as a tool-augmented RS assistant.

Source: Remote Sensing

DOI: 10.3390/rs18101613

Authors: Xuan Zhou, Xuefeng Wei, Zhi Qu, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Institution: Nara Institute of Science and Technology

Language: English

Topic: Multimodal Machine Learning Applications

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $3,258

Funder: China Scholarship Council