Measuring AI Scientists: From Exams to Discovery
Year: 2026
Type: preprint
Abstract: Large language models and agentic systems are increasingly embedded across the scientific work-flow, from literature synthesis and hypothesis generation to code execution, data analysis and writing. This broadening of use exposes a mismatch in evaluation: current benchmarks can test whether models answer difficult scientific questions, but not whether they can advance science itself. Discovery depends on sequences of decisions under uncertainty, in which hypotheses must be made testable, interventions executed reliably, and imperfect evidence interpreted with calibrated restraint. In this Perspective, we argue that AI scientists should be evaluated through discovery episodes rather than isolated problems. A discovery episode records the evolving scientific state, the actions taken, the observations produced, the revisions made and the provenance needed to reproduce and audit the process. This framing separates three coupled capabilities—hypothesis generation, computational or experimental execution, and result interpretation—and motivates evaluations that score trajectories as well as outcomes, preserve failures and null results, and distinguish genuine discovery from rediscovery. Such evaluations are central for developing AI scientists that contribute responsibly to trustworthy scientific discovery.
Source: ChemRxiv
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7203676309