Year: 2023
Type: dissertation
Abstract: Neural Ordinary Differential Equations (NODEs) [Che+18] have improved accuracy and memory efficiency over general deep neural networks but suffer from the vanishing gradient problem and the large number of function evaluations (NFEs) problem. In chapter 3, we proposed Heavy Ball NODEs (HBNODEs), leveraging the continuous limit of classical momentum-accelerated gradient descent, to improve NODEs training and inference. We show that HBNODEs has two major advantages: (1) The adjoint of HBNODEs also satisfies Heavy Ball ODEs, accelerating both forward and backward solvers; (2) the spectrum of HBNODEs is well-structured so that HBNODEs is capable of learning long-term dependencies from long time series. In chapter 4, we proposed Graph Neural Diffusion with a Source Term (GRAND++), a class of NODEs on graphs, for deep learning on graphs with a limited number of labels. We study the limiting behavior of GRAND++ and show that it does not converge to a constant even when the depth goes to infinity. We provide experiments to show that GRAND++ can provide accurate classification even when the number of labels is limited. In chapter 5, we study how proximal implicit solvers can improve NODEs training in some scenarios. We show that for stiff NODEs, proximal implicit solvers have smaller NFEs than commonly used explicit solvers, and thus speed up training.
Author: Xia, Hedi
Institution: University of California, Los Angeles
Language: English
Subfield: Statistical and Nonlinear Physics
Field: Physics and Astronomy
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: Algorithms, Theory, and Validation of Deep Graph Learning with Limited Supervision: A Continuous Perspective, Collaborative Research: ATD: Robust, Accurate and Efficient Graph-Structured RNN for Spatio-Temporal Forecasting and Anomaly Detection, Collaborative Research: Differential Equations Motivated Multi-Agent Sequential Deep Learning: Algorithms, Theory, and Validation, TRUSTWORTHY PHYSICS-INFORMED DEEP LEARNING FOR PREDICTIVE SCIENTIFIC COMPUTING, Collaborative Research: ATD: Robust, Accurate and Efficient Graph-Structured RNN for Spatio-Temporal Forecasting and Anomaly Detection, FRG: Collaborative Research: Robust, Efficient, and Private Deep Learning Algorithms
OpenAlex ID: W7043821694