Work

Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels

Year: 2021

Type: preprint

Abstract: We study the relative power of learning with gradient descent on differentiable models, such as neural networks, versus using the corresponding tangent kernels. We show that under certain conditions, gradient descent achieves small error only if a related tangent kernel method achieves a non-trivial advantage over random guessing (a.k.a. weak learning), though this advantage might be very small even when gradient descent can achieve arbitrarily high accuracy. Complementing this, we show that without these conditions, gradient descent can in fact learn with small error even when no kernel method, in particular using the tangent kernel, can achieve a non-trivial advantage over random guessing.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2103.01210

Authors: Malach, Eran, Kamath, Pritish, Abbe, Emmanuel, Srebro, Nathan

Institutions: Hebrew University of Jerusalem, Toyota Technological Institute at Chicago, École Polytechnique Fédérale de Lausanne

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W3135746662

Cites: 21

Cited by: 4

Related to: 10