DIFT: Division- and Inverse-Square-Root-Free Fixed-Point Approximation of Transformer Normalization for Hardware-Oriented Inference
Year: 2026
Type: article
Abstract: On resource-constrained edge hardware, fixed-point Transformer inference is constrained by GEMM kernels and non-GEMM normalization paths. In LayerNorm and Softmax, division, inverse square root, and denominator reciprocal operations complicate fixed-point datapath design. We propose DIFT (Division and Inverse-Square-root-Free Transformer normalization), a fixed-point scheme that removes division and inverse square root from the targeted normalization paths. Starting from a trained checkpoint, we keep the weights, matrix multiplications, and Softmax exponentials unchanged, replacing only the LayerNorm inverse-standard-deviation path and Softmax denominator reciprocal path with Q7I2 fixed-point approximations. Calibration remains separate from final evaluation. We evaluate fine-tuned BERT-base on four GLUE tasks with three seeds and fixed-length-block perplexity (PPL) of GPT-2 small on WikiText-2. LayerNorm-only, Softmax-only, and Joint (DIFT) settings isolate the error sources. For BERT, the mean accuracy change under Joint (DIFT) is approximately −0.24 to −0.08 percentage points. GPT-2 test PPL increases from 29.009 to 31.092, a ΔPPL of +2.083. Under this protocol, the same approximation affects token-level-likelihood PPL more strongly than classification accuracy. The results provide an algorithm-level assessment before hardware implementation on edge NPUs, FPGAs, or custom ASICs. Synthesis and hardware measurements of latency, area, power, and energy are outside the scope of this study.
Source: Electronics
Authors: Keke Wei, Takashi Kusaka, Takayuki Tanaka
Institution: Hokkaido University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7208764972