Benchmark-path fidelity in parallel robot-policy evaluation: a CALVIN case study
Year: 2026
Type: article
Abstract: Evaluating long-horizon robot manipulation policies is a high-throughput computing workload: one CALVIN-style campaign sweeps checkpoints, seeds and task sequences across many concurrent simulator workers, consuming tens of thousands of rollouts before any score is inspected. Evaluation-path correctness is therefore a workflow-validation concern in parallel evaluation: an adapter defect shared by workers can yield a well-formed table of zeros while scheduler and determinism checks pass. We report a CALVIN case study and use it to propose the Benchmark-Fidelity Diagnostic Protocol (BFDP): a cheap-first set of single-worker controls, run before a parallel sweep, that audits export horizon, teacher replay semantics and reset fidelity in a fixed order. On Composing Actions from Language and Vision (CALVIN) scene D with a vision-language-to-diffusion pipeline, BFDP separates three findings of three different kinds. For the two tasks with a measured effect onset, the fixed 8-step export ends before the observed onset, and across benchmark-critical tasks it retains only 12–16% of the mean teacher-episode length. This is a confirmed defect of the export path: across all 14 episodes, in a paired intervention with export horizon as the only changed factor, exact-reset teacher replay rises from 0/14 to 14/14. The symbolic path returns one canonical robot state for all 64 probed initial conditions; this is the benchmark’s documented neutral reset rather than a defect, and the resulting exact-versus-symbolic split measures the sensitivity of our open-loop teacher control to its start state. Exact continuous replay reaches 1/3 on an official three-subtask probe, which bounds what teacher replay can establish under the official chained protocol rather than indicating a further defect. The full set of controls is between 0.36 and 1.8% of the subtask-rollout workload of the campaign it precedes, depending on how far the policy gets before each official sequence terminates. We present BFDP as a proposed diagnostic checklist demonstrated on one benchmark, one scene and one policy family, not as a validated general standard.
Source: The Journal of Supercomputing
Authors: Yuanzhi He, Qiming Wang, Jiayu Xu
Institution: Cardiff University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,290
OpenAlex ID: W7223474091