Context Embeddings for Efficient Answer Generation in Retrieval-Augmented Generation
Year: 2025
Type: conference-paper
Abstract: Retrieval-Augmented Generation (RAG) allows overcoming the limited knowledge of LLMs by extending the input with external information. As a consequence, the contextual inputs to the model become much longer slowing down decoding time affecting the time a user has to wait for an answer. We address this challenge by presenting COCOM, an effective context compression method, reducing long contexts to only a handful of Context Embeddings, speeding up the generation time by a large margin. Our method allows for different compression rates, trading off decoding time for answer quality. Compared to earlier methods, COCOM allows for handling multiple contexts more effectively, significantly reducing decoding time for long inputs. Our method demonstrates an inference speed-up of up to 5.69 times while achieving higher performance compared to existing efficient context compression methods
Institutions: University of Amsterdam, The University of Queensland
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4407953545