Work

Generating and choosing research questions with AI agents

Year: 2026
Type: preprint
Abstract: Judgements of candidate research questions are noisy, and practitioner checklists such as FINER have not been tested against research outcomes. We tested methods for generating and choosing research questions in one field, the observable behaviour of small open language models, where every question could be piloted in minutes. Three Claude models generated questions by four methods (168 after merging near-duplicates). Four selection methods scored every question before any outcome was known; every question then had a full pilot on three small models and a fixed literature search. The primary outcome was a new effect detected in a pilot: the full pilot met a pass test fixed in advance, and the search found no earlier report of the effect in the same direction. Of 168 questions, 37 (22%) gave one; an audit after the results found earlier reports for 8 of these 37, without changing the comparison of selectors. Critique and forecast (two models named each question's most likely flaw, then three models forecast its outcome) predicted this outcome better than chance (area under the ROC curve, AUROC, 0.684; 95% interval 0.59 to 0.78); in an exploratory analysis, almost all of this came from the forecast that the pilot would pass. A screening pilot, a short pilot built independently of the full one, did about as well (0.712) and predicted passing better (0.807 against 0.622). A FINER checklist scored by one model (0.514) and random scores (0.550) did not beat chance. Plain prompting and transfer from distant fields gave the most new effects (28% and 27% of their questions) and gap mapping from recent abstracts the fewest (8%), mostly because its earlier reports were abstracts its generators had read; rates also differed by generating model (19% to 33%), though these differences could be chance. The method built from these results generates with several models, screens every question with a short pilot, and checks novelty with a recorded search. The components were tested separately in one field, Claude models did all the generating and judging, and a new effect need not be important. Where pilots are cheap, the results favour running short pilots of candidate questions over rating them, and searching the literature over asking models whether a question is new.
Language: English
Open Access status: green
OpenAlex ID: W7226019017
Cites:
Cited by:
Related to: 10