Work

An Analog and Digital Hybrid Attention Accelerator for Transformers with Charge-based In-memory Computing

Year: 2024

Type: conference-paper

Abstract: The attention mechanism is a key computing kernel of Transformers, calculating pairwise correlations across the entire input sequence. The computing complexity and frequent memory access in computing self-attention put a huge burden on the system especially when the sequence length increases. This paper presents an analog and digital hybrid processor to accelerate the attention mechanism for transformers in 65 nm CMOS technology. We propose an analog computing-in-memory (CIM) core, which prunes 75% of low-score tokens on average during runtime at ultra-low power and delay. Additionally, a digital processor performs precise computations only for 25% unpruned tokens selected by the analog CIM core, preventing accuracy degradation. Measured results show peak energy efficiency of 14.8 and 1.65 TOPS/W, and peak area efficiency of 976.6 and 79.4GOPS mm2in the analog core and the system-on-chip (SoC), respectively.

DOI: 10.1109/esserc62670.2024.10719540

Authors: Ashkan Moradifirouzabadi, Divya Sri Dodla, Mingu Kang

Institution: University of California San Diego

Language: English

Topic: Advanced Memory and Neural Computing

Subfield: Electrical and Electronic Engineering

Field: Engineering

Domain: Physical Sciences

SDG: Industry, innovation and infrastructure

Open Access status: green