Empirical Scaling Laws of Dither Noise Suppression in Foundation Models: From Sub-1B to 27B Parameters Under Quasi-Subtractive TPDF Quantization
Year: 2026
Type: preprint
Abstract: Post-training quantization (PTQ) of Large Language Models (LLMs) to sub-8-bit representations introduces severe systematic distortion when performed via deterministic Round-to-Nearest (RTN). Although classical signal processing establishes that Triangular Probability Density Function (TPDF) dithering strictly linearizes quantizers and eliminates harmonic quantization spurs, early experiments on sub-billion parameter models reported degraded perplexity due to additive dither variance (Delta^2 / 4). In this work, we reveal and rigorously prove the fundamental Scaling Law of Dither Noise Suppression: across layer projection dimension D, Central Limit Theorem (CLT) noise averaging suppresses additive dither variance as O(D^{-1/2}), while deterministic RTN bias accumulates coherently across depth L as O(L). Across an empirical evaluation ladder of modern hybrid State-Space Model (SSM) and Transformer foundation models (the Qwen3.5/Qwen3.8 family spanning D=1024 to D=5120, L=24 to L=64, and 0.8B to 27B parameters), we demonstrate that for D >= 4096, non-subtractive TPDF inherently outperforms RTN in directional representation preservation. Furthermore, we invent Quasi-Subtractive TPDF Quantization, an out-of-core architectural mechanism that dynamically reconstructs pseudo-random dither sequences in compute registers from deterministic counter-based PRNG seeds (Philox-2x32), algebraically canceling dither variance back to the theoretical minimum of Delta^2 / 12 without caching full-precision weights. In Qwen3.5-9B, Quasi-Subtractive TPDF achieves an unprecedented 95.8% reduction in cumulative directional drift relative to RTN (8.97e-6 vs. 2.13e-4). On the flagship 64-layer Qwen3.8-27B model executed via layer-by-layer out-of-core streaming within 4.2 GB VRAM on consumer hardware, Quasi-Subtractive TPDF preserves a cosine fidelity of 0.997852 against uncompressed FP16. Finally, we demonstrate that TPDF provides dual auxiliary benefits: enabling 4.0x long-context Key-Value (KV) cache compression with zero attention entropy degradation up to 4096 tokens, and inducing intrinsic stochastic gradient masking that diminishes adversarial white-box jailbreak gradient alignment from 0.9920 to 0.9674.
Author: Tarkov, Yaroslav Aleksandrovich
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7233603200
Cites:
Cited by:
Related to:
