Evaluator Reliability Under Selection Pressure: Implications for Agentic Test-Time Search
Year: 2026
Type: preprint
Abstract: This paper investigates the reliability of learned evaluators under increasing Best-of-N selection pressure using fixed candidate pools generated by three language models across GSM8K and HumanEval. The study evaluates how selected-response correctness changes across increasing selection budgets while independently measuring task correctness. It identifies evaluator-dependent differences in selection reliability, including cases where increasing evaluator-score margins coincide with declining task accuracy. On GSM8K, DeepSeek PRM-selected accuracy for Llama 3.1 8B Instruct peaks at 80.4% at N = 8 before declining to 72.2% at N = 256, despite increasing evaluator-score margins. On HumanEval, CodeScaler demonstrates significant advantages over Skywork-Reward for Llama 3.1 8B Instruct and Mistral 7B Instruct, although this advantage is not established for Qwen3 14B after multiple-comparison correction. The findings motivate independent correctness audits across multiple selection budgets when evaluating learned evaluators for use in test-time search and agentic systems. Autonomous agents are not directly evaluated in this study. Workshop acceptance: Accepted for poster presentation at AIAS+ 2026: The 3rd Workshop on Lifelong Agents: Learning, Aligning, and Evolving (LLA). The author was unable to participate in the in-person poster session. Research conducted independently by the author.
Author: Pradeep, L
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225864166
