How Many Bits Does a Model Weight Need? The Delta-Squared-Over-Twelve Law for Quantised Regression Weights, Tested on Six UCI Data Sets With Nearest Rounding, Stochastic Rounding and Subtractive Dither
Year: 2026
Type: preprint
Abstract: Storing model weights in fewer bits saves memory and energy, and the classical model of quantisation noise says what it costs: rounding to a grid of step Delta adds an error of variance Delta^2/12 to each weight, so the expected increase in squared prediction error is Delta^2/12 times the trace of the feature second-moment matrix, and each extra bit cuts it by a factor of 4. I test this on random-Fourier-feature ridge regressions with 2,048 weights fitted to six public UCI data sets, quantising the weights to 2 to 12 bits with three rules. Subtractive dither makes the law exact for any weights, and I prove it; across bit widths from 2 to 12 the mean over data sets of its quadratic term is between 0.88 and 1.10 times the prediction. Stochastic rounding is unbiased but doubles the error variance, which I prove for uniformly placed weights and observe (1.99 to 2.03 times Delta^2/12 from 4 bits up). Ordinary nearest rounding matches the law from about 5 bits but is 6 to 25 times worse at 2 to 4 bits, because the rounding error then copies the sign pattern of the weights. A formula for the bits needed to keep the test error increase within 1% matches dither at all six data sets and nearest rounding to within 2 bits (4 to 10 bits predicted). Two further results explain why single quantised models scatter around the law: the quadratic term's relative standard deviation is sqrt(2 tr(H^2) - 1.2 sum H_jj^2)/tr(H), about 0.9 here because the effective rank of the features is only about 2.5, and the test-set gradient adds a cross term that shrinks only by a factor 2 per bit and overtakes the quadratic term at about 8 bits.
Author: Aguru, Manoj Kumar
Institution: Chaitanya
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225380594