Efficient Many-Shot In-Context Learning with Task-Level Compression and Instance-Level Retrieval
Year: 2026
Type: conference-paper
Abstract: Many-shot in-context learning (ICL) lets large language models exploit hundreds or thousands of demonstrations, but its cost grows with the prompt.We argue that many-shot demonstrations carry two kinds of information: task-level information, such as task structure, which emerges only across many examples, and instance-level information, such as exact inputoutput associations and rare exceptions, which resides in individual demonstrations.Existing efficiency methods commit all information to a single placement: demonstration retrieval keeps a few demonstrations as raw text, while context compression places the full pool into a compact memory.We study where each kind of information should be placed.On ManyI-CLBench, each one-sided method falls short on tasks that depend on the information it discards, while combining the two retains much of the full many-shot performance at a much smaller inference budget.Synthetic tasks further show that compressed memory and retrieved examples can complement each other but sometimes interfere.
Authors: Masayuki Kawarada, Ukyo Honda, Sho Takase
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7225628708