Year: 2020
Type: conference-paper
Abstract: Transformers have proved effective in many NLP tasks.However, their training requires non-trivial efforts regarding carefully designing cutting-edge optimizers and learning rate schedulers (e.g., conventional SGD fails to train Transformers effectively).Our objective here is to understand what complicates Transformer training from both empirical and theoretical perspectives.Our analysis reveals that unbalanced gradients are not the root cause of the instability of training.Instead, we identify an amplification effect that influences training substantially-for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output.Yet we observe that a light dependency limits the model potential and leads to inferior trained models.Inspired by our analysis, we propose Admin (Adaptive model initialization) to stabilize the early stage's training and unleash its full potential in the late stage.Extensive experiments show that Admin is more stable, converges faster, and leads to better performance 1 .W (V 2 ) h , where f s is the row-wise softmax function and W (•) h are parameters.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1198008555, G1472423617, III: Small: Multi-Dimensional Structuring, Summarizing and Mining of Social Media Data, III: Medium: Collaborative Research: Mining and Leveraging Knowledge Hypercubes for Complex Applications, G3688555466, G4713059963, G4734753682, BIGDATA: F: Collaborative Research: Taming Big Networks via Embedding, G8175839138, III: Medium: Collaborative Research: StructNet: Constructing and Mining Structure-Rich Information Networks for Scientific Research
OpenAlex ID: W3103334733