The Attention Dispersion Law and Dual-Path Decoupling in Autoregressive Transformers
Year: 2026
Type: preprint
Abstract: Large language models (LLMs) excel at smooth semantic continuation, yet their behavior remains vulnerable when maintaining discrete invariants—exact integer counting constraints over long sequences. Across a pre-registered benchmark of 1,416 canonical probes (1,320 in the core Argon v8-v16 series, plus 96 in replication v6) evaluated under zero-temperature on modern Mixture-of-Experts transformer runtimes (with strict No-Tools isolation across all 960 purely cognitive counting probes in v9 and v11-v16, alongside tool-manipulation protocols in v8 and v10), we investigate the mechanistic locus of the discrete counting error (the N -> N+1 fencepost overshoot). Through a sequence of anatomical interventional experiments, we demonstrate that:1. Exoneration of the Decoder: By applying an algebraic amplifier (f(N) = 10N - 6, v15) to separate attention scan drift (Delta = +/-10) from output unembedding glitches (Delta = +1), we find strictly 0 / 96 (0.0%, two-sided 95% Clopper-Pearson CI [0.0%, 3.8%]) decoder off-by-one errors, while input perceptual scan drift accounts for 34.4% (33/96).2. Disqualification of the Tokenizer Hypothesis: By byte- and character-matching a repetitive Korean subword marker (12 bytes UTF-8) against a clean Unicode geometric box-drawing block (12 bytes UTF-8, v16), we show that replacing language with geometry fails to eliminate phantom errors (35.4% vs 41.7%, Fisher exact test p = 0.675). This fails to support the strong BPE tokenization hypothesis and is consistent with the hypothesis that the discrete counting limit reflects an attention phase boundary constraint.3. Dual-Path Attention Decoupling: In adversarial gaslighting probes (v13), the model follows instructions to emit a falsehood in its visible content (93.8%), yet spontaneous "truth leaks" directly from the latent reasoning channel (thinking) into the terminal token occur in 2/48 probes. This demonstrates that terminal logits can decouple from local autoregressive n-gram context. We formalize these empirical findings into the Attention Dispersion Law: as monotonic sequences exceed local attention span capacity (N >= 8..16), positional embedding phase interference smears receptive boundaries, inducing discrete overshoot. Consequently, deterministic agent architectures should complement in-context data passing with external reference passing and deterministic runtime parity gates. The Russian version of this preprint is the second file in this record:main_PACKAGE_v2_ru.pdf. / Русская версия — второй файл записи:main_PACKAGE_v2_ru.pdf.
Author: Borovikov, Vladimir V.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223829162