Year: 2026
Type: conference-paper
Abstract: Recent reports suggest that LLMs can handle increasingly long contexts.However, many existing benchmarks for context understanding embed substantial query-irrelevant content, which shifts evaluation toward retrieving relevant snippets rather than fully integrating all provided information.Under this setting, we view that current benchmarks can overestimate true context-understanding ability of LLMs.In particular, we demonstrate that when the context consists entirely of query-relevant text, even advanced models such as GPT-4o fail to reliably integrate inputs as short as 200 tokens.To evaluate this capability more rigorously, we introduce NEEDLECHAIN, a benchmark designed to test whether models can faithfully incorporate all given evidence.NEEDLECHAIN includes three variants that differ in the required order of comprehension, along with a parallel benchmark based on the needle-ina-haystack(NIAH) paradigm.By comparing these variants, NEEDLECHAIN enables a more comprehensive assessment of context understanding.We further propose a training-free strategy that encourages models to reflect all available information, ROPE contraction, highlighting the importance of full-context integration and pointing to new directions for improving reliable reasoning over context. Mixed Chain Backward Chain Independent Needle:-{A} received 320$ last week Dependent Needle:-{A} earns twice as much as {B} -{A} earns half as much as {B} -{A} earns the same salary as {B} Forward ChainQuestion:How much salary did {D} get? {A} received 320$ last week {B} earns twice as much as {A} {C} earns half as much as {B} {D} earns twice as much as {C} {D} earns twice as much as {C} {C} earns half as much as {B} {B} earns twice as much as {A} {A} received 320$ last week {D} earns twice as much as {C} {A} received 320$ last week {C} earns half as much as {B} {B} earns twice as much as {A} NeedleChain NeedleStack (NIAH) {B} received 320$ last week {D} received 160$ last week {C} received 640$ last week
Authors: Hyeonseok Moon, Heuiseok Lim
Institutions: Korea University, Korea University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166863176