Work

The Orchestrator Adds Nothing on Recovery: A Pre-registered Paired Study of LangGraph versus a Native Loop on Success, Cost, and Exactly-Once Resume

Year: 2026
Type: preprint
Abstract: Agent frameworks promise durable, resumable agents, and the promise is rarely tested against a plain hand-written loop under matched controls. This study pre-registered the test. Two arms run the same 12 tasks with the same deterministic scripted policy, the same tools, the same step executor, and the same budgets, paired by seed: one arm sequences the work with LangGraph 1.2.20 and its SQLite checkpointer, the other with a hand-written loop over its own SQLite event log. The scripted policy is a declared scope limit: the study measures the orchestration layer only (sequencing, durability, recovery, re-execution, cost, code surface), not model quality. On task success the arms are equivalent with evidence: overall pass^1 is 0.979 in both arms (94 of 96 runs each), the mean paired per-task difference is 0.000 with a 90 percent confidence interval of [0.000, 0.000], inside the registered equivalence margin of plus or minus 0.05 (H1: Supported). Consistency (pass^k, k = 1 to 8) is identical in the two arms at every k. On cost, policy invocations are identical (median 5.0 per run in both arms, ratio 1.000 against a registered bound of 1.05) and LangGraph's median wall time per run is lower in these implementations (0.293 seconds against 0.980, ratio 0.299 against a bound of 1.10), a comparison the paper flags as implementation-sensitive because the native arm commits state synchronously after every step (H2: Supported). On recovery after a SIGKILL (48 fault runs per arm, two kill classes), both arms pass every run with every intended side effect applied exactly once and no duplicates: the exactly-once resume rate is 1.000 in both arms, so the registered tie form holds and the orchestrator adds nothing measurable on recovery in this design (H3: Supported, tie form). The difference is in what recovery costs: in the step-boundary kill class, LangGraph recomputed exactly one already-finished step in all 24 runs (its durable checkpoint lags the last finished step), while the native event log recomputed none in all 24; LangGraph also needed less arm-specific code (81 against 96 code lines) and a durable state store about 2.25 times larger (median 36,864 against 16,384 bytes per run). Exactly-once came from the idempotency discipline both arms applied, not from the orchestrator. The debugging axis (H4) is reported as Not yet measured, for the reason registered in advance: the arms persist traces in incommensurable units, so the registered record-count procedure would measure storage format, not diagnosis effort.
Language: English
Open Access status: green
OpenAlex ID: W7235931591
Cites:
Cited by:
Related to: 10