Measuring the Fairness Gap Between Retrieval and Generation in RAG Systems using a Cognitive Complexity Framework
Year: 2025
Type: conference-paper
Abstract: In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.
Institution: Amazon (United States)
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4412378037