Where the √u Floor Comes From and When It Matters: Direct Deficit Estimation and a Conformance Test for Similarity Measures
Year: 2026
Type: preprint
Abstract: Plain Language Summary When you compare a neural network with a re-run of itself, the similarity score comes out as something like 0.9999999999. Papers usually report "one minus that" as the amount of change. A computer, however, stores 0.9999999999 with a limited number of digits, so "one minus that" cannot be trusted below about 10⁻¹⁶. This is a limit of the arithmetic, not of CKA or RSA. A rearranged formula gives the tiny true number directly, without ever building the number c close to one. For typical precision changes in a real network, both routes give the same answer. They part ways in three places: on a bitwise-identical re-run, where the true change is exactly zero but the usual formula reports made-up drift; in float32 computations, where the usual formula breaks on every real activation cell whose perturbation is near 10⁻⁷ and on 66.4% of cells at 10⁻⁴–10⁻³, while ours stays usable above about 10⁻⁶; and in public software packages, where five of forty-two applicable checks return small non-zero values at the identity. Abstract Researchers often measure how much a network's internal representation has changed by computing 1 − s, where s is a similarity score such as CKA that sits very close to one. Subtracting two numbers that agree to machine precision throws away most of the digits, so the smallest change this recipe can detect is set by the arithmetic, not by the similarity measure. We show that for every cosine-form similarity the change can be computed directly as 1 − s = ½ ∥Â − B̂∥²_F. This form never computes s, can never be negative, and returns exactly zero when the two inputs are bitwise identical. The identity is classical for canonical correlations (Knyazev and Argentati, 2002); we apply it to linear CKA, RBF CKA, and Pearson RSA, which have no principal-angle interpretation. Against a 60-digit mpmath reference, the usual two-step computation is up to 6.7×10¹⁵ times less accurate than the direct form at a perturbation of 10⁻¹⁴, where it simply returns 0.0. Whether this matters depends on the setting. We test it on 459 perturbed cells and 255 bitwise-identical controls from 17 text corpora on GPT-2, with additional runs on SmolLM2 and Pythia. In float64, the two methods agree to within 2.328 × 10⁻² for realistic execution changes, so the direct form only adds headroom. It makes a real difference in three cases. First, on bitwise-identical re-runs, the usual formula reports drift on 185 of 255 cases (72.5%) where the true change is zero; the direct form returns exactly zero every time. Second, when the analysis runs in float32, the usual formula fails on every cell with δ ≈ 10⁻⁷ and on 66.4% of cells at 10⁻⁴–10⁻³, while the direct form stays usable above δ ≈ 10⁻⁶ (median relative error 1.999× 10⁻⁵). Third, the precision used to save the activations can limit accuracy before the similarity formula ever does. We also derive a closed-form expression, the resolution Hessian, for the smallest detectable change in linear CKA. Note: A 4-page extended abstract titled "Where the √u Floor Comes From and When It Matters: A Conformance Audit of Similarity-Measure Estimators" was accepted to the UniReps 2026 Workshop (Extended Abstract Track). This version extends it with experiments on real activations (17 corpora, a float32 analysis arm, seed drift, near-duplicate pairs), the resolution Hessian, and the detection-limit framework.
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225621153