Work

Associative Recurrent Memory Transformer

Year: 2024

Type: preprint

Abstract: This paper addresses the challenge of creating a neural architecture for very long sequences that requires constant time for processing new information at each time step. Our approach, Associative Recurrent Memory Transformer (ARMT), is based on transformer self-attention for local context and segment-level recurrence for storage of task specific information distributed over a long context. We demonstrate that ARMT outperfors existing alternatives in associative retrieval tasks and sets a new performance record in the recent BABILong multi-task long-context benchmark by answering single-fact questions over 50 million tokens with an accuracy of 79.9%. The source code for training and evaluation is available on github.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2407.04841

Authors: Rodkin, Ivan, Kuratov, Yuri, Bulatov, Aydar, Burtsev, Mikhail

Institutions: Moscow Institute of Physics and Technology, AIRI - Artificial Intelligence Research Institute, London Institute for Mathematical Sciences

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Analytical Center for the Government of the Russian Federation, Moscow Institute of Physics and Technology

Awards: G3736275146, G5927019889, G6108322314

OpenAlex ID: W4400517511

Cites:

Cited by:

Related to: 10