Get started
Work

Mathematical Theory of the Transformer, Volume I: Foundations, Rationality, and the Incremental Transformer

Year: 2025
Type: preprint
Abstract: Mathematical Theory of the Transformer — Volume I develops, in a single sustained argument, a geometric and dynamical theory of self-attention, analyses the training dynamics of BERT as a concrete case study, and derives from those analyses a family of rationalised and incremental architectures that determine their own structure from the data. CENTRAL THESIS The architectural choices of the standard Transformer — number of heads, head dimension, residual stream, layer normalisation, feed-forward width — are not independent hyperparameters to be tuned by trial and error. They are tied to a small number of geometric and dynamical quantities that can be measured, diagnosed, and controlled. Once those quantities are identified, the architecture itself becomes a derived object: it can be rationalised at design time and grown incrementally during training. STRUCTURE OF THE VOLUME The book is organised in four parts and 20 chapters, plus 5 appendices and a "Guide to the Main Concepts" in the front matter. Part I (Chapters 1–3): Sequences, embeddings, and positional encoding; the self-attention mechanism; BERT as a running case study. Part II (Chapters 4–9): The symmetric/antisymmetric decomposition; post-softmax persistence; the eight canonical regimes of attention; canonical signatures and attention geography; reading signature geography; signature re-extraction and global information flow. Part III (Chapters 10–14): Motor formalism on the Lie algebra so(d); self-attention motor decomposition; signature orbits and training dynamics; sharpness, activation, and silence; redundancy and design principles. Part IV (Chapters 15–20): The Rationalised Transformer; INCRT I (core dynamic rationalisation) and II (extensions, guarantees, applications); spectral gate dynamics; experimental validation; Anchored Multi-Head Bilinear Gating (A-MHB-G) as the architectural synthesis of the volume. STATUS OF THE RESULTS The manuscript distinguishes throughout between four categories of statement and keeps them separable chapter by chapter: Proved results: theorems, propositions, lemmas, and corollaries with explicit hypotheses and complete proofs. Empirical findings: statements supported by experimental evidence on BERT-base or on the dedicated case studies (SARS-CoV-2 variant classification, proteome MLM pre-training, Euler–Lagrange adaptive control, geopolymer inverse design), reported with multi-seed aggregates and effect sizes. Conjectures: mathematically precise statements not yet proved within the volume. Architectural design principles: prescriptions translating the diagnostic results into architectural choices. MAIN MATHEMATICAL CONTRIBUTIONS Rank-Neutrality Theorem for layer normalisation (Chapter 1). Symmetric/antisymmetric decomposition of the QK bilinear form and the Cycle Circulation Theorem (Chapter 4). Eight canonical regimes of attention as a Frobenius-projection coordinate system (Chapter 6). Global signature geography and re-extraction principle (Chapters 7–9). Motor formalism and self-attention motor decomposition (Chapters 10–11). Canonical Symplectic (Darboux) Decomposition of the antisymmetric sector and four-class Fixed-Point Classification of training orbits (Chapter 12). Sharpness, activation, and silence as diagnostic coordinates on signature space (Chapter 13). INCRT (Incremental Transformer) controller with compressed-sensing-style guarantees and finite-termination theorem (Chapters 16–17). Spectral gate dynamics: degeneracy regimes and the unified gate (Chapter 18). COMPANION PAPERS The volume cites and builds on a set of companion preprints by the author and collaborators: The Geometry of BERT (Bonino, Ghione, Cirrincione 2025) — arXiv:2502.12033. Rank, Head-Channel Non-Identifiability, and Symmetry Breaking (Cirrincione 2026) — arXiv:2604.23681. On the Geometry of Positional Encodings in Transformers (Cirrincione 2026) — arXiv:2604.05217. Hierarchical Kernel Transformer (Cirrincione 2026) — arXiv:2604.08829. INCRT: An Incremental Transformer That Determines Its Own Architecture (Cirrincione 2026) — arXiv:2604.10703. Temporal Attention for Adaptive Control of Euler–Lagrange Systems (Cirrincione & Fagiolini 2026) — arXiv:2605.06877. Submitted manuscripts on protein transformers (Cirrincione & Lovino), geopolymer design (Cirrincione & Grassia), and runtime-incremental control (Cirrincione & Fagiolini). AUDIENCE AND PREREQUISITES The book is written for graduate students and researchers in machine learning, applied mathematics, and theoretical computer science. The reader is assumed comfortable with linear algebra at the graduate level, the rudiments of differential geometry and Lie algebras, and the language of modern deep learning. No prior exposure to the Transformer is assumed: Chapters 1–3 set up the architecture and the BERT case study from scratch. RELATION TO VOLUME II Volume II will extend the architectural programme along three axes: the dynamic dictionary controller (DDCL); the integration of INCRT with prototype-based learning; and the extension to generative regimes (autoregressive language models, diffusion-time task families, in-context learning). The present volume does not depend on these extensions: every theorem, every empirical result, and every architectural prescription in Volume I is self-contained. References to Volume II are forward signposts, never premises. CHANGES SINCE VERSION 1.0 Chapter 7 has been split into Chapter 7 (Global Analysis: Canonical Signatures and Attention Geography — theory) and Chapter 8 (Reading Signature Geography: Observables, Diagnostics, and Limits — empirical and diagnostic uses). The book grows from 19 to 20 chapters. New Appendix E (Extended Protocols and Deferred Directions for Signature Geography), collecting the reporting protocols, dictionary expansion, dependency map to later chapters, and the sheaf-theoretic direction that was not pursued. Preface expanded with the central thesis, the four-category classification of results (proved / empirical / conjectural / design principles), and an explicit delimitation of the relation between Volume I and Volume II. New "Guide to the Main Concepts" page in the front matter, mapping the eight proprietary concepts (signature, geography, motor, orbit, silence, rationalised Transformer, INCRT, A-MHB-G) to chapter and role. Approximately 170 unresolved cross-references resolved; all chapter-level references now point to canonical labels. Bibliography consolidated to approximately 190 entries, including all companion papers with arXiv identifiers, submission venues, and standard external references on attention, mechanistic interpretability, optimisers, mixture-of-experts, growing networks, and in-context learning. Editorial language normalised: removed residual "draft", "corrected", and "Removed" markers in chapter and section titles; calibrated potentially over-strong claims; replaced hardcoded chapter numbers with cross-references throughout. TOC, List of Figures, and List of Tables formatting: widened number column to eliminate spacing collisions for deep section numbers and chapter-numbered floats. Front-matter copyright page reformulated as an unpublished-manuscript notice with persistent archive (DOI) reference. Caption shortening across the volume: long captions split between short forms (in the List of Figures) and full explanations (under each figure), producing a much tighter front-matter index of figures. LICENSE AND CITATION This is an unpublished book manuscript. All rights reserved by the author. The persistent archival version is deposited on Zenodo; the concept DOI always resolves to the latest version. Citations are welcome and should reference both the title and the Zenodo version (or concept) DOI.
DOI: 10.48550/arxiv.2502.12033
Language: English
Open Access status: green
OpenAlex ID: W4407759049
Cites:
Cited by:
Related to: 10