Get started
Work

The conservation of inductive bias: every learner is a code, and generalization is its length

Year: 2026
Type: preprint
Abstract: Machine learning lacks a single quantity that is defined for every learning algorithm, measurable on real training pipelines, and sufficient to state what the algorithm can and cannot learn. We argue that such a quantity exists: the implicit code length, the cumulative log loss that a learner incurs when it predicts each example before training on it. We develop this idea into a small, self-contained theory with complete proofs. First, a conservation law: for every learner and every set of inputs, the implicit code lengths of all possible labelings satisfy Kraft's equality, so inductive bias is a fixed budget that can be moved between tasks but never increased; we give a version for noisy tasks. Second, a representation theorem: every learner is exactly a Bayesian predictor for its own implicit prior over labelings. Third, an identity rather than a bound: under independent sampling, the expected implicit code length equals the sum of the test risks along the learning curve, so the learning curve is the discrete derivative of the code length. From it we derive constraints on plateaus, on power-law scaling exponents, and on the area of any double-descent bump. Fourth, an exact decomposition of the cost of in-context learning into the mutual information between task and data plus a pretraining gap. Most ingredients are classical; the contribution is their synthesis into one accounting system, and a research programme built on it. We state three conjectures about deep networks that the theory makes precise, explain why a naive simplicity conjecture must fail for computational reasons, and specify a laptop-scale protocol that would refute each conjecture. The paper is purely theoretical and reports no experiments.
DOI: 10.5281/zenodo.24141713
Institution: University of Pavia
Language: English
Open Access status: green
OpenAlex ID: W7225824313
Cites:
Cited by:
Related to: