Work

Eliminating Harmonic Quantization Artifacts in Deep Neural Networks via Triangular Probability Density Function (TPDF) Dithering

Year: 2026
Type: preprint
Abstract: Post-Training Quantization (PTQ) to sub-8-bit representations (INT6, INT4) is critical for reducing the memory footprint and compute intensity of large-scale Transformer architectures. However, conventional uniform quantization almost universally employs deterministic Round-to-Nearest (RTN) rounding. While high-precision regimes permit treating quantization error as benign additive white Gaussian noise, this assumption catastrophically breaks down below 8 bits. In sub-8-bit regimes, deterministic RTN generates non-linear, signal-correlated harmonic distortion that coherently accumulates across deep residual blocks and exponentiates within Softmax attention operators, triggering attention entropy degradation and singular spectrum distortion. In this work, we present a rigorous theoretical foundation and empirical framework for non-subtractive Triangular Probability Density Function (TPDF) dithering in deep neural network tensor operations. By analyzing the Fourier expansion of the quantizer error characteristic and applying Schuchman's condition within the non-overload granular region, we provide complete analytical proofs that: (1) injecting 2-LSB peak-to-peak TPDF dither guarantees an identically zero conditional error expectation (E[e|w] = 0, for all w), rendering quantized General Matrix Multiplications (GEMM) unbiased estimators of full precision and eliminating first-order O(Delta) task-loss gradient bias; and (2) TPDF dither establishes an identically constant conditional error variance (Var[e|w] = Delta^2/4, for all w) by virtue of the double zeros of the sinc^2 characteristic function at harmonic frequencies, completely eliminating noise power modulation. In contrast, Uniform (RPDF) dither suffers from persistent signal-dependent variance ripple of 0.25*Delta^2. We further establish exact non-saturation guard-band conditions, analyze static frozen versus dynamic runtime dither regimes, and prove orientation-independent angular preservation in self-attention. Empirically, TPDF dithering improves Spurious-Free Dynamic Range (SFDR) by +16.6 dBFS over RTN at INT4, eliminates systematic input-error cross-correlation (reducing empirical rho from -0.0882 to -0.0050, statistically indistinguishable from theoretical zero at N=32,768), reduces 32-layer systematic directional bias drift by 63.0%, and prevents attention entropy collapse in Transformer blocks.
Language: English
Open Access status: green
OpenAlex ID: W7230295033
Cites:
Cited by:
Related to: 10