Excess Description Length: An Information Measure for Generalizable Structure Learned from Finite Data
Year: 2026
Type: conference-paper
Abstract: Learning algorithms transfer predictive structure from data into model parameters, yet existing information-theoretic quantities primarily characterize either dataset complexity or asymptotic learning costs. We introduce excess description length (EDL), a compute-indexed measure of the generalizable information extracted by a learning algorithm from finite data. EDL admits an operational interpretation via prequential coding: it is the excess codelength incurred on first exposure to data relative to a predictor whose expected log-loss equals the final model’s population loss, thereby excluding memorization effects on the training set. We establish that EDL satisfies fundamental properties of an information measure—non-negativity, additivity under product structure, monotonicity in compute, and a processing bound on extractable information—while explicitly decoupling the roles of data (information source) and computation (extraction mechanism). A finite-data saturation bound formalizes that compute cannot extract more generalizable information than exists in the data. We show that EDL penalizes overfitting through the generalization gap, distinguishing genuine learning from memorization. These results establish EDL as a rigorous foundation for quantifying information transfer in learning systems operating on finite datasets. An extended version appears at arXiv:2601.04728.
Author: Elizabeth Donoway
Institution: University of California, Berkeley
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7204187224