Work

How GUI Agents Receive Instructions: An Exploratory Mapping of 85 Benchmark Families

Year: 2026
Type: preprint
Abstract: Preprint. An exploratory mapping of how 85 GUI and computer-use agent benchmark families give the agent its task, coded for a declared, non-exhaustive frame of 101 candidates (a keyword-targeted stratum and a seeded random comparison sample). The paper separates three constructs that descriptions of benchmarks often mix: how the goal is specified, what the agent observes while acting, and how completion is judged. In both strata the goal is given as natural-language text in almost every component; a video, recording or demonstration is part of the specification in 9 of 81 components of the targeted stratum and 1 of 66 of the comparison stratum, and more components are coded as using video as authoring input (18 of 81 in the targeted stratum, some thinly evidenced) than as a specification channel. All coding was done by AI agents and audited by a second AI pass; no independent human recoding was performed, and the study was not preregistered. Counts describe the sample, not the field. Competing interest: The author is affiliated with Vertov (https://vertov.app), a company developing screen-recording and AI-agent collaboration software. Vertov has a commercial interest in the subject of this research. Companion dataset: see related identifiers.
Language: English
Open Access status: green
OpenAlex ID: W7232107759
Cites:
Cited by:
Related to: 10