Attention Society: Three Implementations and a Negative Result for the Hypothesis that an Internal Economy Accelerates Neural Network Training
Year: 2026
Type: preprint
Abstract: We report, as a negative result, three successive implementations of the Attention Society hypothesis: that nodes of a neural network which hold a currency (Attention Pay, AP) and compete, invest, and standardize can reach a target performance with less time and compute than ordinary attention training. In Version 1, AP at the level of FFN neurons only modulated output gains and could not change the actual connectivity; wall-clock time to target was 39–55% slower than the baseline. In Version 2, AP became a compute entitlement (a share of the token slots processed by each mixture-of-experts (MoE) block), and the correlation between AP share and actual load reached 0.91–0.97 (Pay and Market variants). Yet on none of the three tasks (sorting, a character-level language model (lm), and continual task switching (shift)) did any variant meet the pre-registered rule for reaching the target in fewer steps than a plain MoE with the same active budget, and wall-clock time was 125–198% slower than the dense baseline. Version 3 followed the designer's revised intent: in a closed economy with no currency issuance (total-money error ≤ 8.5e-13 in all 129 runs), money buys the right to emulate the computation of the block that best fits the reward. All 129 runs of Version 3 finished, and the hypothesis is rejected within the scope of these experiments (three seeds; only effects of about 10% or more are detectable). Steps to target did not differ, under the pre-registered rule, from MoE, from dense, from emulation without money, or from investment in random targets (for the key comparison against emulation without money: sort +4.3%, lm −1.3%, shift −2.2%), and wall-clock time was 37.4% (sort) and 34.5% (lm) slower than MoE. A variant that leaves the transformer unmodified (the head variant) also showed no step difference from dense and was 92.0% (sort) and 62.2% (lm) slower in wall-clock time. The economy beat emulation without money in one setting only, the head variant with weight interpolation (lerp) on sort (−36.0%, 3/3 seeds), because the comparison arm collapsed in diversity and failed to reach the target within budget on all three seeds; this is not a win over dense. We propose four candidate explanations for the absence of a speed-up (the first three were not isolated by separate tests): backpropagation already optimizes each block, so "following the leader" makes blocks resemble one another rather than adding information; the choice of whom to follow made no detectable difference; in a closed economy money pools in a few holders and degenerates into a schedule that weakens emulation; and settlement overhead. We close with a list of designs that later work should avoid.
Author: Park, Seonyul
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225313169