Right Answers, Wrong Habits: Selecting Self-Generated Agent Trajectories by Evidence Sufficiency
Year: 2026
Type: preprint
Abstract: Tool-using agents are routinely improved by fine-tuning on their own successful trajectories, and “successful” almost always means that the final answer was correct. But a correct answer need not be justified by what the agent observed. On a closed-world ProofWriter tool-use task, a sound checker certifies that the observed evidence determines the answer in only about one in seven of a 4B agent’s correct trajectories, and 316 of 877 outcome-selected training trajectories submit before observing anything. Fine-tuning the agent on them barely helps: the student often answers without looking and scores below an agent that gives the same answer to every question. We instead keep a correct trajectory only if a deterministic evidence-sufficiency certificate, which replays the agent’s tool calls, shows that the observed replies determine the answer under the declared semantics. With the same model, trajectory source, episode count and update count, certificate selection lifts macro accuracy from 39.7% to 59.3% (+19.6 points, 95% CI [18.4, 20.8], positive in all five seeds; our pre-specified primary contrast) and beats a length control, an observation-ratio heuristic and an LLM judge; against a judge told the task’s semantics the comparison is inconclusive. The advantage replicates in smaller and larger students of the same family (+22.4 and +28.7 points) and, more weakly, in a second model family (+4.4 points, 95% CI [3.0, 5.8]). Students remain weak on question types their teacher never certified, and all results come from one synthetic benchmark.
Author: Xu, Song
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: green
OpenAlex ID: W7234511282
