Beyond task success: domain-grounded evaluation of Earth-observation agents in environmental risk workflows
Year: 2026
Type: article
Abstract: AI agents now run Earth observation (EO) workflows end to end, and their authors judge them on the task: answer accuracy, tool use, report scores and case studies. None of these evaluations asks whether the agent acted on a bad input. We ran five published EO and climate agents (LLM-Geo, Earth-Agent, WildfireGPT, ClimSight and PangaeaGPT) on their authors' own example tasks, and all five solved them on correct data. We then corrupted one input per system with a fault any expert would reject at a glance: impossible values (land at −267 °C, a July of 86.77 °C, 35 °C in Arctic water) or the wrong place (Tennessee census tracts for a North Carolina task, a wildfire site in the open Atlantic). Across 106 runs, scored on what reached the user, 85 used the bad data and said nothing, 4 said the data were wrong, and 17 changed what they delivered. In several runs the agent's own trace shows it noticed the problem and delivered an answer anyway. The faults are defined against four checks an expert applies to EO data: Validity, Fit, Uncertainty and Compliance. This study tests the first two. We propose evaluating EO agents by corrupting their data on purpose, reading the trace, and scoring what the agent did about the fault. Poster presented at the BIFOLD-ESA Workshop "Agentic AI for Earth Observation", Berlin, 19 October 2026.
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7221859080
FWCI:
Cites:
Cited by:
Related to: