Work

The Positivity of the Neural Tangent Kernel

Year: 2025

Type: article

Abstract: . The Neural tangent kernel (NTK) has emerged as a fundamental concept in the study of wide neural networks. In particular, it is known that the positivity of the NTK is directly related to the memorization capacity of sufficiently wide networks, i.e., to the possibility of reaching zero loss in training via gradient descent. Here we will improve on previous works and obtain a sharp result concerning the positivity of the NTK of feedforward networks of any depth. More precisely, we will show that, for any nonpolynomial activation function, the NTK is strictly positive definite. Our results are based on a novel characterization of polynomial functions, which is of independent interest.

Source: SIAM Journal on Mathematics of Data Science

DOI: 10.1137/24m1659534

Authors: Luís Carvalho, João L. Costa, José Mourão, Gonçalo Oliveira

Institutions: Iscte – Instituto Universitário de Lisboa, University of Lisbon, Instituto Superior Técnico, Centro de Análise Matemática Geometria e Sistemas Dinâmicos

Language: English

Topic: Neural Networks and Applications

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: diamond

Funder: Fundação para a Ciência e a Tecnologia

Awards: Centro de Análise Matemática, Geometria e Sistemas Dinâmicos, Centro de Análise Matemática, Geometria e Sistemas Dinâmicos