Discovery Needs a Hypothesis: Separating Hypothesis Proposal from Search in Structure Discovery by Language Models with Code Interpreters
Year: 2026
Type: preprint
Abstract: Does structure discovery fail for want of search, or for want of a hypothesis to search for? We separate the two in a family of fully controlled worlds: 32 states with a hidden partition into four blocks of eight, a small library of operators shared by every block, and "roles" that apply one operator per block. A model sees a table of roles and must predict 24 hidden cells, with a penalty for confident errors. Given the table alone, GPT-6 with a code interpreter solves 0 of 24 titles; given the same table plus one paragraph describing the hypothesis, it solves 23 of 24 with no confident errors, answering exactly the cells the hypothesis determines (23 is the ceiling; Claude Opus goes from 0 to 17). Search is not what is missing: ordered to write and run a program without the paragraph, GPT-6, Opus and Sonnet execute code on every title and still solve 0/24; told only that the cells are determined, GPT-6 searches six times longer and solves 8/24, with errors; given the hypothesis as one of three candidates, it picks it and solves 21/24 with none. The paragraph has two halves. The first (states fall into blocks, roles map blocks to blocks) the model states unprompted in 24 of 24 blind tables; the second (the same operator recurs on every block) it never discovers, and that half alone carries the effect: 1/24 without it, 20--23/24 with it in any of five wordings. The second half is not supplied by gradient descent in a network built around the right structure (0/24), nor induced from one table (0/24) or from six tables under "find the common law" (0/12); under "reverse-engineer the generating program" it is occasionally written (3/12; a second model, 0/6), and once written the model's own law is as good as ours (6/6 on new tables, 36/36 rejections on worlds where it is false). In this family the bottleneck is not the solver but the hypothesis the solver is handed.
Source: Preprints.org
Author: Shilin Yin
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226003980
