The Start of the Window: RoPE's Critical Dimension Predicts the Base Frequency Needed for 128K–256K Context Extension
Year: 2026
Type: article
Abstract: Technical report, raw per-trial evaluation results, launch scripts and analysis code on far-position (depth-0) retrieval failure after RoPE base-frequency (ABF) context extension of a 9B multilingual OpenEuroLLM model to 128K and 256K tokens. A controlled theta sweep shows that the failure is fixed by the RoPE base, not by long-context data, and the critical-dimension theory of RoPE extrapolation, computed from the pretraining configuration alone, predicts the critical base at 64K, 128K and 256K. The PDF is in paper/rope-base-depth0.pdf.
Author: Moëll, Birger
Institution: AI Sweden
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236044129
