Work

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum

Year: 2024

Type: preprint

Abstract: We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression (KRR) in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from previous work; for exponential decay, our bound is non-trivial and novel. Our contribution is two-fold: (i) we rigorously prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption, closing an existing gap in the literature; (ii) we identify that the independence of the features plays an important role in guaranteeing tempered overfitting, raising concerns about approximating KRR generalization using the Gaussian design assumption in previous literature.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2402.01297

Authors: Cheng, Tin Sum, Lucchi, Aurelien, Kratsios, Anastasis, Belius, David

Institutions: University of Basel, Vector Institute, McMaster University, UniDistance Suisse

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W4391591572

Cites:

Cited by:

Related to: 10