Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information
Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) have demonstrated an impressive ability to retrieve and summarize complex information, but their reliability in conflicting contexts remains poorly understood.We introduce an adversarial extension of the Needle-in-a-Haystack framework in which three mutually exclusive "needles" are embedded within long documents.By systematically manipulating factors such as position, repetition, layout, and domain relevance, we evaluate how LLMs handle contradictions.We find that models almost always fail to signal uncertainty and instead confidently select a single answer, exhibiting strong and consistent biases toward repetition, recency, and particular surface forms.We further analyze whether these patterns persist across model families and sizes, and we evaluate both probability-based and generation-based retrieval.Our framework highlights critical limitations in the robustness of current LLMs-including commercial systems-to contradiction.These limitations reveal potential shortcomings in RAG systems' ability to handle noisy or manipulated inputs and exposes risks for deployment in high-stakes applications.
Authors: Murathan Kurfali, Robert Östling
Institution: Stockholm University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Vetenskapsrådet
OpenAlex ID: W4416035030