Year: 2023
Type: preprint
Abstract: Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of training Transformers by carefully studying a simple yet canonical linearized shallow Transformer model. Specifically, we train linear Transformers to solve regression tasks, inspired by J.~von Oswald et al.~(ICML 2023), and K.~Ahn et al.~(NeurIPS 2023). Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics. Consequently, the results obtained in this paper suggest that a simple linearized Transformer model could actually be a valuable, realistic abstraction for understanding Transformer optimization.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, National Research Foundation, National Research Foundation of Korea, Korea Advanced Institute of Science and Technology, Ministry of Science and ICT, South Korea, Samsung, Office of the Secretary of Defense, Institute for Information and Communications Technology Promotion, Office of Naval Research
Awards: G1769687045, G1808747736, CAREER: Modern nonconvex optimization for machine learning: foundations of geometric and scalable techniques, G2148870006, G2926380509, AI Institute for Learning-Enabled Optimization at Scale (TILOS), G445402534, G5415505637, G6391819523, G6671297155, G7064264939, G7450053111, G8876996369, G981443286
OpenAlex ID: W4387322684