Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation
Year: 2026
Type: conference-paper
Abstract: RAG evaluations often rely on citations or retrieved evidence traces for correctness checks, provenance claims, and audits, implicitly assuming that evidence remains reproducible under routine retrieval settings.We test this assumption in a controlled diagnostic study where queries, embeddings, and decoding are fixed while retrieval depth, chunk size, and overlap vary.We call the resulting change in attributed evidence retrieval jitter and measure evidence identity at two levels: document (doc_id) and exact cited span (doc_id, span_hash).Across BEIR ArguAna and Sci-Fact, we observe a consistent Stability Gap: document overlap remains moderate while span overlap often collapses, including many cases of total span turnover despite non-empty retrieval.We interpret span-level instability as a diagnostic of exact evidence-trace reproducibility, not semantic equivalence.These findings motivate reporting stability diagnostics alongside citation-based evaluation metrics for more reproducible evaluation practice.
Author: Punitha Ponnuraj
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166829402