Year: 2025
Type: paratext
Abstract: In the span of less than one year, AI agents have moved from being a research curiosity to underpinning the largest software platform updates in a generation.Agents promise to streamline substantial portions of knowledge work and progress has been swift, with early-2025 work by METR suggesting a potential doubling of task complexity every seven months.In this talk, we submit this claim to deeper empirical scrutiny: we examine what it takes to build good benchmarks of agentic performance on long-running knowledge work tasks.We review recent contributions, many by the ServiceNow Research team, on task-space dimensions ranging from browser use and multimodal understanding, to data analytics and deep research.We also touch on benchmarks of agentic safety and security, suggesting that their evaluations cannot be decoupled from primary task performance.We arrive at a more nuanced conclusion than earlier results, with major problems in the field yet to be solved by frontier agents.
Authors: Ehsan Kamalloo, Nicolas Gontier, Xing Han Lu, Nouha Dziri, Shikhar Murty, Alexandre Lacoste
Institutions: ServiceNow (United States), Stanford University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Australian National University
OpenAlex ID: W4412944452