Rational Synthesizers or Heuristic Followers? Analyzing LLMs in RAG-based Question-Answering
Year: 2026
Type: conference-paper
Abstract: Retrieval-Augmented Generation (RAG) is the prevailing paradigm for grounding Large Language Models (LLMs), yet the mechanisms governing how models integrate groups of conflicting retrieved evidence remain opaque.Does an LLM answer a certain way because the evidence is factually strong, because of a prior belief, or merely because it is repeated frequently?To answer this, we introduce GroupQA, a curated dataset of 1,635 controversial questions paired with 15,058 diverselysourced evidence documents, annotated for stance and qualitative strength.Through controlled experiments, we characterize grouplevel evidence aggregation dynamics: Paraphrasing an argument can be more persuasive than providing distinct independent support; Models favor evidence presented first rather than last, and Larger models are increasingly resistant to adapt to presented evidence.Additionally, we find that LLM explanations to group-based answers are unfaithful.Together, we show that LLMs behave consistently as vulnerable heuristic followers, with direct implications for improving RAG system design.We release the dataset and code for reproducibility: https://github.com/anaphade/GroupQA
Author: Atharv Naphade
Institution: Carnegie Mellon University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166827489