Attention is My Memory: Positional Encoding-free Length Invariance in Transformers
Year: 2026
Type: preprint
Abstract: This paper introduces a positional encoding-free transformer augmented with a single content-addressed memory from DABSN, enabling extreme length generalization without positional embeddings or context-index limits. Trained on 256-token sequences, the model achieves 96.9% accuracy on multi-query associative recall at 64× training length, 100% accuracy at 8× memory load, and 96–100% passkey retrieval accuracy at one million tokens (4096× training length). The approach also matches positional transformers on character-level language modeling at training length. Across 459 experimental runs, the study demonstrates how a lightweight memory interface can overcome major limitations of positional addressing in transformer-based retrieval.
Author: Nicholas Rosdahl
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Keywords: associative recall, memory retrieval
Open Access status: green
OpenAlex ID: W7234083906
