Work

RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies

Year: 2026
Type: conference-paper
Abstract: Overview of RoboLab.RoboLab addresses large-scale evaluation of real-world policies via a novel evaluation framework.By introducing a streamlined pipeline for generating new scenes and tasks (top row), RoboLab enables rapid extensibility for testing generalization capabilities.The accompanying benchmark, RoboLab-120, introduces various evaluation axes and a suite of metrics and analysis tools, and demonstrate benchmark-level correlation with real-world benchmarks (bottom row).Abstract-The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid performance saturation and a lack of true generalization testing.Existing benchmarks often exhibit significant domain overlap between training and evaluation, trivializing success rates and obscuring insights into robustness.We introduce RoboLab, a simulation benchmarking framework designed to address these challenges.Concretely, our framework is designed to answer two questions: (1) to what extent can we understand the performance of a real-world policy by analyzing its behavior in simulation, and (2) which factor most strongly affect policy behavior.First, RoboLab enables human-authored and LLM-enabled generation of scenes and tasks in a robot-and policy-agnostic manner within a highfidelity simulation environment.We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes: visual, procedural, relational, across three difficulty levels.Second, we introduce a systematic analysis of real-world policies that quantify both their performance and the sensitivity of their behavior to controlled perturbations, exposing significant performance gap in current state-of-the-art models.By providing granular metrics and a scalable toolset, RoboLab offers a scalable framework for evaluating the true generalization capabilities of task-generalist robotic policies.
Language: English
Open Access status: bronze
OpenAlex ID: W7154270010
FWCI: 156.1
Cites: 32
Cited by: 73
Related to: 10