Work

Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

Year: 2022

Type: preprint

Abstract: We introduce a modified version of the excess risk, which can be used to obtain tighter, fast-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for $[-1, 1]$-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around $0$. The primary new technical tool is a novel result for sequences of interdependent random vectors which may be of independent interest. We empirically evaluate these new bounds on a number of real-world datasets.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2210.11289

Authors: Biggs, Felix, Guedj, Benjamin

Institution: University College London

Language: English

Topic: Machine Learning and Algorithms

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Agence Nationale de la Recherche, Engineering and Physical Sciences Research Council

Awards: UKRI AI Centre for Doctoral Training in Foundational Artificial Intelligence, A PAC-Bayesian Representation Learning Perspective, PAC-Bayesian Agnostic Learning

OpenAlex ID: W4307078152

Cites:

Cited by: 1

Related to: 10