Work

Preregistration: nortrebenk v1.0 pilot measurement (P3)

Year: 2026
Type: article
Abstract: nortrebenk measures how well and how fast coding agents solve tasks taken from public code in Norwegian public agencies, and what each solved task costs. The tasks are merged pull requests from the agencies' public GitHub repositories. The ground truth is the agency's own tests: a task counts as solved when the tests that failed before the change pass, and the tests that passed still pass. Status: preregistered; the pilot measurement has not been run. The preregistration is frozen at git tag prereg-v1 (PREREG.md). No agent has been run on a task in the set. AI disclosure: implemented with Claude Code.
Language: English
Open Access status: green
OpenAlex ID: W7235566177
FWCI: 2.197
Cites:
Cited by: 1
Related to: 10