How Do Agent Benchmarks Test Underspecified Instructions? An Evidence Map of Definitions, Injection, Clarification and Scoring, 2022 to 2026
Year: 2026
Type: preprint
Study design: Systematic Review
Abstract: Preprint. Many agent benchmarks now test what a system does when an instruction leaves something out or admits more than one reading, and each defines the problem in its own way. This evidence map codes 215 benchmarks and datasets dated 2022 to 2026, located by searches frozen on 19 November 2026, on how they define underspecification, construct it, label it, let the agent respond, simulate the user, score the outcome, and release the data. Most are recent: 93% are dated 2025 or 2026. 66% state a definition of the problem they test. 67% create the missing information by injecting it into existing tasks or by generating tasks, and 13% use natural or mixed origins. The agent may ask a question in 71%; where it asks, a language model usually plays the user. Task success is the most common metric (75%), but 144 benchmarks also report a metric outside the common families, and only 95 state how many tasks are underspecified. Resolving the gap needs a visual or other non-text referent in 5 benchmarks according to their sources and in 14 when coder inferences are counted; it is not reported in 22. A data license is stated for 50%. Blind second coding of a seeded sample agreed with the initial coding on 90% of origin codes. The map describes the works located by these searches, not the field. AI-use disclosure: Claude (Anthropic) assisted with literature discovery, screening, data extraction, classification, analysis and manuscript preparation. The research methods and quality-control procedures are described in Section 4. The author reviewed the output.
Author: Shrestha, Ankur
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236651503
