UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities
Year: 2026
Type: conference-paper
Abstract: Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries.However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus.In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address.To address this, we introduce UniversalRAG, an anyto-any RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities.Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modalityaware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis.Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query.We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines. * Equal contribution; † Equal advising(A) RAG with Single Modality (D) UniversalRAG (Ours) (C) RAG with Single CorpusText+Image+Video Corpus Image Corpus Retrieve Generate Text+Image RAG Query: Describe features of Camellia japonica.Bicycle wheels are typically designed … (B) RAG with Single Granularity Video Corpus Full video(2h) Short Video RAG Query: How did Messi score last game?Video RAG Query: How can I replace a bike wheel?Simple Query: What is the capital of France?
Institution: Korea Advanced Institute of Science and Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4417299405