Work

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Year: 2018

Type: preprint

Abstract: The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.1802.10026

Authors: Garipov, Timur, Izmailov, Pavel, Podoprikhin, Dmitrii, Vetrov, Dmitry, Wilson, Andrew Gordon

Institutions: Skolkovo Institute of Science and Technology, Cornell University, National Research University Higher School of Economics

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Ministry of Education and Science of the Russian Federation, Samsung

Awards: G6349798659, RI: III: Medium: Scalable Machine Learning for Automating Scientific Discovery in Astrophysics

OpenAlex ID: W2788838181

Cites: 21

Cited by: 332

Related to: 10