Impact of chunking granularity on accuracy and token consumption in retrieval-augmented generation for question-answering
Year: 2025
Type: dissertation
Abstract: This research investigates the role of document chunking in retrieval-augmented generation systems. Even though RAG addresses LLM drawbacks such as knowledge cutoffs and errors, performance is impacted by the techniques employed to divide documents into searchable sections, and these, particularly concerning chunk granularity, are often underexplored. Conventional chunking techniques can fragment context or overlook complex details within documents, which challenges the balance between retrieval accuracy and token consumption. This research systematically investigates how chunking choices affect RAG system accuracy and token efficiency in question-answering, aiming to inform the design of more resource-conscious RAG systems that consider operational and environmental costs. The research employs a quantitative, experimental design to compare fixed-size, sentence-aware, and hybrid chunking, each applied at target maximum chunk granularities of 256, 512, and 1024 tokens. It employs twenty 10-Q financial reports and an existing 195-pair question-answer dataset from GitHub. Two RAG pipeline architectures are implemented and evaluated, a baseline NaiveRAG dense retrieval system and an advanced graph-based PathRAG system. While the number of retrieved tokens measures efficiency, a multifaceted evaluation system evaluates answer quality using a wide range of lexical and semantic similarity indicators. Findings indicate that 512-token chunks generally balance between accuracy and context for both RAG systems, with 512-token methods often proving effective for NaiveRAG. The advanced PathRAG system also effectively utilizes concise 256-token chunks, especially fixed-size ones for specific metrics and tabular data. Conversely, large 1024-token chunks consistently diminish accuracy across all configurations. Increasing the chunk size from 256 to 512 tokens often inflates retrieved token counts by over half and sometimes more than doubles them. This increase in consumed tokens leads to modest accuracy gains, occasionally by two percentage points or less. While preferred granularity also shifts with question and source type, further token escalation with 1024-token chunks consistently degrades overall performance.
Source: Aaltodoc (Aalto University)
Author: Lensu, Urho
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6982336101