Within-Process Determinism Is Not Reproducibility: A Measurement Hazard in Behavioural Comparison of Language Models
Year: 2026
Type: preprint
Abstract: We report a measurement hazard encountered while comparing the behavioural effect of in-context conditioning against explicit low-rank weight updates in a 1.7B-parameter language model. Repeated measurement of an identical reference condition within a single process returned bit-identical values on every repetition — a spread of exactly zero. The same quantity, measured in separate processes with identical seeds, prompts, and code, varied by 22.5%. An identical rank-256 adapter measured 0.0624 in one process and 0.0881 in another. Normalising each measurement by a reference measured in the same process did not remove the discrepancy but enlarged it from 41% to 77%, because the reference and the treatment drifted in opposite directions. We estimated the noise floor twice from within-process repetitions and revised it upward twice; both estimates were of the wrong quantity. A power-law fit and a derived crossover built on cross-process data were retracted, and single-process re-measurement moved the crossover from an extrapolated 214–298 to a directly observed 128. Enabling PyTorch's deterministic-execution switches does not remove the effect: across six further processes the across-process spread was 15.9% with the switches off and 11.9% with them on, against a within-process spread of exactly zero in all six. Reloading the model inside a process does not perturb the result at all, which locates the differing state at process initialisation rather than at model load. We quantify the phenomenon, give the diagnostic that distinguishes it from ordinary stochastic noise, and give five reporting practices that would have caught it. The general point is that a noise floor is valid only if its repetitions span the sources of variation the floor is used to license — so the most precise available estimate can be the most misleading one.
Author: Hu, Xiangyu
Language: English
Subfield: Information Systems and Management
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7203634627
Cites:
Cited by:
Related to: