Year: 2020
Type: preprint
Abstract: Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}\left(N\right)$, where $N$ is the sequence length. We show that this formulation permits an iterative implementation that dramatically accelerates autoregressive transformers and reveals their relationship to recurrent neural networks. Our linear transformers achieve similar performance to vanilla transformers and they are up to 4000x faster on autoregressive prediction of very long sequences.
Source: arXiv (Cornell University)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Paläogeographische Entwicklung des helvetischen Ablagerungs- raumes während der Kreide und des Tertiärs (Teilgesuch ZH: Jura/Kreide-Grenze und Barrème-Apt in der Ostschweiz), Analyse palynologique et paléobotanique des interstades Würmiens et des derniers Interglaciaires dans deux bassins Pré-Alpins de Suisse occidentale, UNISON: Universal Structure-Aware Output Learning for Natural Language Understanding and Generation
OpenAlex ID: W3034573343
