Understanding the mechanism of pretraining stabilization heuristics: A variance-oriented perspective
Year: 2024
Type: article
Abstract: Language model pretraining has been breaking the glass ceiling for various natural language processing tasks and has been viewed as one of the most significant successes of deep learning, continuously challenging our understanding of learning and cognition.Recently models, including GPT-4 and BART, fueled by an unprecedented scale of computing and data, exhibit unprecedented intelligence, that some even refer to as "sparks of artificial general intelligence".The success of large-scale pretraining hinges on intricate engineering heuristics.While the empirical benefits of these heuristics are evident, their underlying mechanisms remain elusive.This dissertation endeavors to demystify the mathematical principles underlying these pretraining heuristics, aiming to illuminate their mechanisms and potentially guide future algorithm developments.Adopting a variance-oriented perspective, my research rigorously inspects the heuristics that are pivotal to the stability of current pretraining practices, emphasizing learning rate warmup, model initialization, and gradient approximation.In this dissertation, I show that these pretraining stabilization heuristics can be coherently elucidated with a unified framework anchored in variance, a classical metric for stability.First, I analyze the variance of adaptive learning rate and model outputs, revealing that both learning rate warmup and model initialization function as variance modulators.Then, I move to explore the variance-bias tradeoff in the discrete variable gradient approximation, i.e., employing a numerical ODE framework, I unveil the underlying dynamics of the approximation bias, achieving secondorder precision with minimal computational overhead.Besides theoretical results, empirical verifications are conducted to verify the assumptions and applicability of the recognized principles.Building upon these insights, this dissertation introduces novel techniques designed to advance the pretraining practices, including RAdam for learning rate warmup, Admin for Transformer model initialization, ReinMax and SparseMixer for gradient approximation.Under the guidance of the recognized principles, all proposed methods require minimal trial-and-error configurations, thereby emerging as robust and high-perform tools for pretraining practices for adaptations.Completing this Ph.D. has been a profoundly challenging and rewarding journey that has spanned several years and consisted of countless hours of hard work and dedication.This monumental task would not have been possible without the support, guidance, and patience of many individuals to whom I owe my sincere gratitude.At the forefront, my deepest appreciation goes to my advisor, Dr. Jiawei Han, for his unwavering support, invaluable advice, and insightful feedback throughout my research.Dr. Han's brilliance and passion have been a constant source of inspiration and have significantly shaped my growth.I always consider myself fortunate to have had the opportunity and flexibility to spend my Ph.D.working on the problems that I believed in, which would not have been possible without Dr.Han's support and guidance.Moreover, Dr. Han instilled in me the wisdom to identify and address the core issues in our field, emphasizing the real solution for real problem on real data.I then extend my sincere thanks to my dissertation committee members, whose insights and suggestions have significantly enriched my research.Prof. Chengxiang Zhai provided valuable advice in my second year, urging me to focus on tackling significant problems rather than solely pursuing challenging ones.Moreover, together with Prof. Jian Peng, he provided me access to GPU resources, making it possible for me to explore deep learning and neural language modeling.I am also appreciative of Prof. Heng Ji, who offered a unique perspective on research, highlighting the balance between academic and industry.My internship experience with Dr. Matthew E. Peters led me to adopt a more deliberate and quality-focused approach to research, prioritizing impactful publications over quantity.I would like to also acknowledge Dr. Jianfeng Gao, a role model whose insightful guidance in research direction has been crucial.His mentorship played a significant role in shaping my research philosophy and approach.A special note of thanks is reserved for my undergraduate advisor, Dr. Linli Xu, who laid the foundation for my research career and kindled my interest in machine learning.I still remember that when our proposed community detection method achieves the state-of-the-art, she encouraged me to not stop at being satisfied by the good performance but to continue explore the underlying mechanism.Her guidance during the critical initial phase of my academic career was instrumental.
Author: Liu, Liyuan
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220764435