The Harness Is the Lab: Measured Self-Improvement for Ontology-Grounded AI Scientists
Year: 2026
Type: preprint
Abstract: Every serious AI-scientist system released in 2026 has converged on the same lesson: generating hypotheses, code and papers is no longer the hard part; deciding which results deserve to be believed is. Karpathy's autoresearch showed that a sealed evaluator, a fixed budget and a git ratchet can run a hundred useful experiments overnight. Sakana's Agentic Scientist, Aster, Recursive's Eureka Machine, Prime Intellect's Prime Agent and Edison's Kosmos each add a missing piece: evidence generation, massive parallelism, a model-harness flywheel, a self-editing harness, and a structured world model. This paper is a position and a blueprint. We argue seven positions, among them that generic coding harnesses are now commodities while task-evolved research scaffolds are not; that research memory must be a typed ontology rather than a chat log; and that any claim of recursive self-improvement without a held-out improvement curve is marketing. We then describe an open research harness that grounds every claim in an ontology of hypotheses, runs and results, keeps its evaluators sealed, and improves itself at three nested levels (search policy, harness skills, model weights), each accepted only on held-out evidence. We define the evaluation protocol we will hold ourselves to, including a self-improvement curve no public system has yet reported. We report no experimental results here; we state falsifiable predictions instead.
Author: Chowdhury, Sayan
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225817478
Cites:
Cited by:
Related to: