Year: 2025
Type: article
Abstract: The Gaussian Error Linear Unit (GELU), a crucial component of the transformer model, poses a significant challenge for hardware implementation. To address this issue, this paper proposes internal symmetry piecewise approximation (ISPA) and error peak search strategy (EPSS) for high-precision and high-efficiency implementation of the GELU activation function. ISPA only approximates the positive axis of the erf in GELU and then leverages its internal symmetry to calculate the negative axis part. With ISPA, the mean square error (MSE) between the fitted result and the true value can reach 4.29×10−9 with 16 parts of the approximation segment, outperforming the regular method, which achieves 1.19×10−6 with 16 parts. Furthermore, EPSS can automatically find suitable and high-precision intervals for our piecewise approximation method. To evaluate the effectiveness of ISPA and EPSS, we conducted experiments on three different ViT models and observed negligible loss of prediction accuracy. The hardware implementation is on an XCZU9EG FPGA running at 450 MHz. Experimental results indicate that ISPA outperforms existing methods.
Source: Electronics
Authors: Jianxin Huang, Yuling Wu, Mingyong Zhuang, Jianyang Zhou
Institution: Xiamen University
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
Award: G3329045930
OpenAlex ID: W4409919053