Work

Making Asynchronous Stochastic Gradient Descent Work for Transformers

Year: 2019

Type: conference-paper

Abstract: Asynchronous stochastic gradient descent (SGD) converges poorly for Transformer models, so synchronous SGD has become the norm for Transformer training.This is unfortunate because asynchronous SGD is faster at raw training speed since it avoids waiting for synchronization.Moreover, the Transformer model is the basis for state-of-the-art models for several tasks, including machine translation, so training speed matters.To understand why asynchronous SGD under-performs, we blur the lines between asynchronous and synchronous methods.We find that summing several asynchronous updates, rather than applying them immediately, restores convergence behavior.With this method, the Transformer attains the same BLEU score 1.36 times as fast.

DOI: 10.18653/v1/d19-5608

Authors: Alham Fikri Aji, Kenneth Heafield

Institution: University of Edinburgh

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Funders: Dell EMC, University of Cambridge, Engineering and Physical Sciences Research Council, Science and Technology Facilities Council

Awards: Peta-5: A National Facility for Petascale Data Intensive Computation and Analytics, G358276009, G6661763476

OpenAlex ID: W2949012179