Work

The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization

Year: 2021

Type: preprint

Abstract: Despite progress across a broad range of applications, Transformers have limited success in systematic generalization. The situation is especially frustrating in the case of algorithmic tasks, where they often fail to find intuitive solutions that route relevant information to the right node/operation at the right time in the grid represented by Transformer columns. To facilitate the learning of useful control flow, we propose two modifications to the Transformer architecture, copy gate and geometric attention. Our novel Neural Data Router (NDR) achieves 100% length generalization accuracy on the classic compositional table lookup task, as well as near-perfect accuracy on the simple arithmetic task and a new variant of ListOps testing for generalization across computational depths. NDR's attention and gating patterns tend to be interpretable as an intuitive form of neural routing. Our code is public.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2110.07732

Authors: Csordás, Róbert, Irie, Kazuki, Schmidhuber, Jürgen

Institutions: University of Applied Sciences and Arts of Southern Switzerland, Dalle Molle Institute for Artificial Intelligence Research, Università della Svizzera italiana, King Abdullah University of Science and Technology

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Nvidia, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Centro Svizzero di Calcolo Scientifico, National Supercomputing Centre Singapore

Awards: Learning neuro-symbolic representations for reinforcement learning, G993971353

OpenAlex ID: W3205643961

Cites: 52

Cited by: 10

Related to: 2