Same Input, Different Experts: Tiny Arithmetic Noise Flips Expert Routing, and a Routing-Conditional Test Cannot See It
Year: 2026
Type: preprint
Abstract: Many AI models send each token, a piece of a word, to 8 of 256 "expert" sub-networks chosen by a router. The correctness tests of the open mixture-of-experts (MoE) kernel benchmark we used check each run against the experts that run picked, so they cannot see the same input picking different experts on different runs. We found a fast multiply whose last-digit noise grew until that happened on every run. We measured one MoE decoder layer on one NVIDIA H200 GPU, with synthetic weights and router (256 experts, 8 chosen per token, FP8 attention inputs). The multiply splits each sum into 7 slices and adds them into a 16-bit (BF16) output in whatever order the hardware finishes. On all 500 replays of the same input, it changed up to 56% of its outputs by one BF16 step (one ulp). Through FP8 rounding and attention, which we did not separate, the difference grew about tenfold. At least one token's expert set then changed on every replay, and the layer output differed by 11 to 17% between replays, yet all 13 correctness tests passed. An FP32, fixed-order multiply cut the difference to at most 0.063%, at a whole-layer cost below our timing resolution. Near-tie flips still remained, even between two copies of identical code. Higher precision shrinks the noise; only sums whose result does not depend on order, everywhere before the router, remove it.
Author: Jani, Harshil
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225338191