Year: 2026
Type: conference-paper
Abstract: Positional encodings are fundamental to Transformers, yet explicit methods like RoPE can degrade under length extrapolation and may incur extra arithmetic and memory-access overhead.In this paper, we propose Scoped Position Encoding (SCOPE), a novel framework that reimagines structured sparsity as an intrinsic position encoding mechanism.Instead of relying on explicit arithmetic signals, SCOPE assigns exponentially scaled look-back scopes to attention heads.We theoretically demonstrate that this simple topological constraint transforms multi-head attention into a hierarchical processor, yielding an order awareness horizon that grows exponentially with depth up to the sequence length.Consequently, SCOPE is parameter-free and avoids relying on fragile positional arithmetic.Empirically, it significantly enhances efficiency by masking the majority of attention computations, offering a theoretical 8× reduction in attention FLOPs at long contexts.Extensive evaluations on LLaMA-3-8B architectures reveal that SCOPE achieves superior native length extrapolation and robust retrieval fidelity compared to RoPE, all while substantially reducing training and inference latency.The code is available at https://github.com/oncemoe/ScoPE.
Authors: Qingguo Qi, Hongyang Chen, Zhao Li
Institutions: Zhejiang Lab, Zhejiang University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G4607961091
OpenAlex ID: W7166867847