When Equivalent Models Receive Different Predictive Scores
Year: 2026
Type: preprint
Abstract: Predictive scores are used to compare candidate programs in automated statistical model discovery. We ask whether a score can change when a program is rewritten without changing its statistical model. In the public scalar-regression evaluator released with SafeStan, rescaling the latent regression coefficient by a constant preserves the exact observation law and yields a normalized finite predictive distribution, yet changes the returned scores while the source, mass and tail checks all pass. In an audit of three authored programs on two tasks, the constant rescaling changes the mean log score by 0.013 and 0.041 nats. A finite-mixture calculation explains the mechanism: integration nodes fixed in a program's own coordinates represent different effective coefficients after reparameterization, and a normalized approximation of this kind can be arbitrarily far from the exact prediction. Transporting the nodes and weights by the standard change of variables removes the discrepancy. Normalization checks should therefore be supplemented by tests of fidelity to the intended predictive law.
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224996304