Year: 2026
Type: article
Abstract: We derive a penalty strength criterion for ridge regression using stochastic complexity, which is a refined variant of the minimum description length principle. Since stochastic complexity does not typically account for the effect of regularization on complexity, despite its ability to simplify models, we are required to make a slight modification to the underlying coding scheme. Our scheme makes use of a weighted ensemble of regularized model fits rather than a mixture of maximum likelihood estimates. Under this modification, regularization is interpreted as reducing model complexity by constraining flexibility. In the case of ridge regression, the complexity penalty term that we derive can be expressed analytically as the log determinant of the residual operator. We demonstrate the effect of this complexity penalty by fitting a linear readout to a reservoir computer, and by performing benchmark testing on publicly available datasets.
Source: Entropy
DOI: 10.3390/e28070735
Authors: Antony Mizzi, David M. Walker, Michael Small
Institution: The University of Western Australia
Language: English
Subfield: Statistics and Probability
Field: Mathematics
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $3,137
OpenAlex ID: W7166638262