Exploration of Low-Energy Floating-Point Flash Attention Mechanism for 18nm FD-SOI CMOS Integration at the Edge
Year: 2024
Type: conference-paper
Abstract: In this work we study the sizing and feasibility of a processing element for the calculation of the attention mechanism used in the Transformer models, using floating point representation. We show that the Flash Attention implementation is needed to enable circuit integration at the edge. Different floating point formats, both standard and non standard representations, were explored and synthesized in 18nm FD-SOI CMOS to compare their area and leakage power. The quality of the results is assessed against a pytorch BERT reference model. The best representation found is a non-standard 12-bit floating point architecture that computes a single block of Flash Attention. Once compared to a RISC- V CPU synthesized with the same technology, shows 104less energy consumption and 103less latency, whilst providing a result with low mean error.
Source: Conference proceedings
Institutions: STMicroelectronics (France), Commissariat à l'Énergie Atomique et aux Énergies Alternatives, CEA Grenoble, Laboratoire d'Électronique des Technologies de l'Information, Centre National de la Recherche Scientifique, Université de Lille, JUNIA, Université Polytechnique Hauts-de-France, Centrale Lille Institut
Language: English
Subfield: Electrical and Electronic Engineering
Field: Engineering
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4402727661