Work

ASDL: A Unified Interface for Gradient Preconditioning in PyTorch

Year: 2023

Type: preprint

Abstract: Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms. In deep learning, stochasticity, nonconvexity, and high dimensionality lead to a wide variety of gradient preconditioning methods, with implementation complexity and inconsistent performance and feasibility. We propose the Automatic Second-order Differentiation Library (ASDL), an extension library for PyTorch, which offers various implementations and a plug-and-play unified interface for gradient preconditioning. ASDL enables the study and structured comparison of a range of gradient preconditioning methods.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2305.04684

Authors: Osawa, Kazuki, Ishikawa, Satoki, Yokota, Rio, Li, Shigang, Hoefler, Torsten

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4375959296

Cites:

Cited by: 1

Related to: 10