Interleaved Rotary Position Embeddings (iRoPE): Mathematical Derivation, Analysis, and Generalization to Infinite Context Length in Transformer Architectures
Year: 2025
Type: preprint
Abstract: Transformer architectures require positional encoding mechanisms to ingest sequences while preserving word order information. However, existing positional embeddings face challenges when extrapolating to sequences longer than those seen during training [Press et al., 2022]. This paper introduces Interleaved Rotary Position Embeddings (iRoPE), a novel positional encoding scheme designed to overcome length generalization limits. Our approach builds on Rotary Position Embeddings (RoPE) [Su et al., 2021], which encode absolute positions via learned rotations to impart relative position dependence in attention. We derive iRoPE from first principles by formulating token position encoding as complex-valued rotations applied in an interleaved coordinate manner across the embedding dimensions. The interleaved formulation leads to elegant mathematical properties: we prove that iRoPE preserves inner-product attention structure and yields attention scores dependent only on relative positional differences, independent of absolute index shifts. Theoretical analysis demonstrates that iRoPE maintains attention stability without divergence as context length grows, exhibiting bounded attention drift even in extreme-length sequences. We further characterize conditions under which iRoPE can generalize to arbitrarily long – effectively infinite – context lengths without loss of fidelity. Finally, we discuss the implications of iRoPE for next-generation long-context LLMs, highlighting that our method introduces no new model parameters and has negligible computational overhead. Through rigorous mathematical proofs and conceptual experiments, this paper lays the foundation for transformer models that train on short sequences yet generalize to extraordinarily long contexts, thereby bridging an important gap towards infinite-context transformers.
Authors: Akbar, Umair, ElShekh, Ahmed
Institutions: University of the Cumberlands, Western Governors University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7080399037