Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models
Year: 2026
Type: conference-paper
Abstract: Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length.Token compression aims to address this challenge by reducing the number of tokens representing inputs.However, existing promptcompression approaches primarily operate in token space and overlook inefficiencies in the latent embedding space.In this paper, we propose K-Token Merging, a latent-space compression framework that merges each contiguous block of K token embeddings into a single embedding via a lightweight encoder.The compressed sequence is processed by a LoRAadapted LLM, while generation remains in the original vocabulary.Experiments on structural reasoning (Textualized Tree), sentiment classification (Amazon Reviews), and code editing (CommitPackFT) show that K-Token Merging lies on the Pareto frontier of performance vs. compression, achieving up to 75% input length reduction with minimal performance degradation.On a Qwen Coder 7B model, 2-Token Merging reduces latency by 20.9% and improves throughput by 27.8% with only a 1.1% relative increase in perplexity.Code is available at https://github.com/shsjxzh/ K-Token-Merging.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7154758010