Year: 2025
Type: preprint
Abstract: Test-time scaling increases inference-time computation through longer reasoning chains and has shown strong performance gains across many domains. However, frontier models still suffer from factuality hallucinations, raising the question of whether increased computation is effective on closed-book knowledge-intensive tasks. In this work, we evaluate 14 reasoning models under different test-time scaling strategies. Our results challenge its effectiveness: increasing test-time computation does not consistently improve accuracy and often leads to more hallucinations. We find that changes in hallucination rates are largely driven by the model's willingness to answer, as longer reasoning encourages more attempts, many of which are incorrect. We also observe patterns consistent with confirmation bias, where extended reasoning reinforces early incorrect beliefs with fabricated details. Finally, we provide an information-theoretic perspective showing that compute-only test-time scaling, as a post-processing procedure of a fixed model, cannot introduce new information about the ground-truth answer, explaining the limited performance gains. Overall, our findings highlight important limitations of current test-time scaling methods for closed-book knowledge-intensive tasks. Code and data are available at https://github.com/XuZhao0/tts-knowledge
Source: arXiv (Cornell University)
Authors: Zhao, James Xu, Hooi, Bryan, Ng, See-Kiong
Institution: National University of Singapore
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4415057168
Cites:
Cited by:
Related to: