The Floor Beneath Learning: Minimum Energy per Useful Bit of Gradient Information
Year: 2026
Type: article
Abstract: Training is priced as information extraction: a stochastic gradient estimate is the true gradient plus isotropic Gaussian noise, so reaching angular error theta requires N = B_noise/theta^2 tokens (McCandlish et al. 2018 noise scale) and yields 0.5*d*log2(1+1/theta^2) reliable bits about the gradient. For the central case (noise scale 1e4 tokens, theta = 0.10 rad, the 405-billion-parameter run of Dubey et al. 2024) one step costs 1.08e6 J at the NVIDIA B200 peak-efficiency bound and yields 1.3483e12 reliable gradient bits: 8.0102e-07 J per reliable gradient bit, 2.79e14 times the Landauer floor at 300 K, and 3.6860e-06 J per bit at whole-job effective efficiency. The price per bit is independent of model size at fixed precision, proportional to the noise scale, and rises steeply with demanded precision. The Landauer floor for the weight update of the same step is 1.86e-08 J. Training energy is therefore dominated by data movement and arithmetic overhead (Horowitz 2014 per-operation energies), not by information physics. Model calculations are labelled SIMULATED, NOT MEASURED. The deposit includes the paper PDF, the computation script (compute.py) that produces every number in the paper, and its outputs (results.json, results.txt).
Author: Aldred, William Alexander
Language: English
Topic: Green IT and Sustainability
Subfield: Electrical and Electronic Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225749613
FWCI:
Cites:
Cited by:
Related to: