Get started
Work

Three Pre-registered Comparisons of pi, Codex CLI and Claude Code on Personal-Assistant Tasks

Year: 2026
Type: preprint
Abstract: When an AI agent fails at a task, the cause can be the language model or the program that runs the model, which is called the harness. Most comparisons of agents change both at once. We compared harnesses while giving them the same model, checked against the provider's own report on every run: the harness pi, configured as one person's assistant, against OpenAI's Codex CLI on OpenAI models (Studies 1 and 2) and against Anthropic's Claude Code on Claude Sonnet 4.5 (Study 3). The tasks were synthetic errands built around ten mistakes that person had repeatedly corrected agents for, such as reporting a job as done without checking it. In Study 1 both harnesses succeeded equally often (98.3% of development runs) and pi used about twice as many tool calls. Study 2 (registered in advance; 30 harder tasks, 540 runs) asked whether that tie hid a ceiling. On a weaker model pi's lead over Codex did not grow (difference in differences -7.8 points, 95% CI [-18.9, 2.2]), an inconclusive result. Pi made 1.84 times as many tool calls on the stronger model only, and removing pi's extensions kept success within a 10-point margin while cutting cost by 17%. Study 3 (registered in advance; the same tasks, 270 runs) found Claude Code and pi neither shown equivalent within 10 points nor shown different (success 43.3% against 52.2%, difference -8.9 points, 95% CI [-23.3, 5.6]). Claude Code was not shown to make fewer tool calls, and removing pi's extensions was neither shown to keep success nor shown to save money. In exploratory analyses Claude Code cost less than pi on 26 of 30 tasks and more on four, among them the three tasks built to overflow the context. Graders that misjudged some answers were corrected after the runs; no registered decision changed.
DOI: 10.5281/zenodo.24045212
Language: English
Open Access status: green
OpenAlex ID: W7225459590
Cites:
Cited by:
Related to: 10