Work

Exploring Generalisation Performance through PAC-Bayes

Year: 2024
Type: dissertation
Abstract: Generalisation in machine learning refers to the ability of a predictor learned on some dataset to perform accurately on new, unseen data. Without generalisation, we might be able to memorise the training data perfectly while predicting poorly on new data, a pathology known as over-fitting. Despite its centrality, the generalisation behaviour of many methods remains poorly understood, particularly in complex domains such as deep learning. Indeed, some models that should over-fit according to traditional theoretical bounds do not. This thesis addresses these issues, particularly in the context of classification, and introduces innovative methods for producing non-vacuous generalisation bounds. \nThe primary thrust is in the development and application of PAC-Bayesian bounds, which are usually used as a method for studying the generalisation of randomised predictors. We begin with an introduction to previous work on the problem of generalisation and to PAC-Bayesian ideas, before applying these in work based on a series of five papers (referenced a-e below). Firstly in (a), we provide lower-variance methods for training stochastic neural networks methods, improving the use of these PAC-Bayes bounds as training objectives. Then, we use PAC-Bayes as a stepping stone to provide non-randomised bounds: (b) using margins, both in general and for several different classifiers; (c) for a specific class of deterministic shallow neural networks (where our bounds are the first to be non-vacuous on real-world data using standard training methods); (d) for majority voting on finite ensembles of classifiers, providing state-of-the-art (and sometimes sharp) guarantees. Lastly in (e), we introduce a PAC-Bayes bound for a modified excess risk, using information about the relative hardness of data examples to reduce variance and tighten a general bound.
Author: Biggs, Felix
Language: English
Open Access status: green
OpenAlex ID: W6991131165
Cites: 113
Cited by: 1
Related to: 10