Attention Is All We Had — But Not What We Needed: Convergent State Machine for Iterative Energy-Based Language Generation
Year: 2026
Type: preprint
Abstract: We introduce the Convergent State Machine (CSM), a novel architecture for language generation that replaces attention entirely with energy-based iterative state refinement. A 150M parameter CSM with zero attention layers matches GPT-2 1.5B on MMLU within 0.3%, using 10x fewer parameters and 13x less training data. State convergence dynamics scale with model size: 66M converges by iteration 15, 150M sustains refinement to iteration 30+. Total training compute: under $50 of A100 GPU time. Keywords: attention-free, iterative reasoning, energy-based, state machine, language model, scaling law
Author: Dwivedi, Arunesh
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162522701
Cites:
Cited by:
Related to: