Year: 2026
Type: conference-paper
Abstract: Long-context language models face efficiency challenges as context lengths expand.Traditional tokenization methods like BPE operate on frequency statistics, ignoring semantic structure and over-tokenizing redundant spans.We propose SemToken, a semantic-aware tokenization framework that adaptively compresses token sequences based on semantic density.SemToken uses lightweight encoders to identify and merge semantically equivalent spans, allocates variable granularity based on local semantic density, and dynamically adjusts token budgets during generation.Evaluations on WikiText-103, LongBench, and BookSum demonstrate 2.4× token reduction, 1.9× inference speedup, and 67% memory reduction while preserving or improving model quality.SemToken integrates seamlessly with existing models and achieves multiplicative benefits when combined with FlashAttention (up to 2.7× total speedup).
Authors: Dong Liu, Yanxuan Yu
Institutions: Yale University, Columbia University
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166871990
FWCI:
Cites:
Cited by:
Related to: