Year: 2026
Type: preprint
Abstract: Recent advances in vision-language-action (VLA) and world-action models (WAMs) have significantly improved robotic manipulation, yet deploying robots in the open world demands a higher level of autonomy beyond executing predefined tasks. Robot agents offer a promising paradigm for this transition, integrating task discovery, contextual reasoning, physical execution, and adaptive recovery into a closed interaction loop. However, existing benchmarks largely evaluate isolated capabilities, failing to capture the complexity of open-world robot autonomy. We introduce RobotAgentBench (RAB), a comprehensive benchmark for evaluating robot agents in realistic, open-world household environments. Through an agent-assisted, human-in-the-loop real-to-sim pipeline, RAB constructs 50 diverse household layouts with interactive objects and physically grounded tasks, encompassing 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 open-ended scenarios. RAB provides a multi-level evaluation of robot agent capabilities: (1) task-level assessments systematically examine five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness; (2) scenario-level evaluation assesses how agents integrate these capabilities to autonomously fulfill high-level user needs. Furthermore, RAB enables systematic comparisons of standalone VLA/WAM models and diverse robot agent architectures under a unified planning-execution-verification framework. Extensive experiments reveal substantial gaps between manipulation and autonomy, demonstrate the advantages of agentic architectures and recursive self-improvement, and highlight the remaining challenges toward open-world robot autonomy.
Source: arXiv (Cornell University)
Authors: Mao, Lingjun, Cao, Jinglin, He, Lukun, Xu, Wenpeng, Yan, Yuchen, Jin, Hengyu, Zhang, Zijun, Tang, Jinzhou, Jin, Songyao, Liu, Ziche, Shao, Yifei, Huang, Junbo, Nan, Fang, Han, Ruobing, Xi, Ziqiao, Xu, Ziming, Liang, Shuang, Zhu, Sibo, Wu, Wenyi, Wang, Xinyue, Huang, Yiming, Zou, Xinkai, Zhu, Gao, Zhou, Kun, Huang, Biwei
Language: English
Subfield: Control and Systems Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223849671