Year: 2024
Type: dissertation
Abstract: Humans display a surprisingly robust ability to generalize their existing knowledge to novel situationsand scenarios, even when they substantially differ from previous experiences. In contrast, deep neural networks can exhibit significantly degraded performance when their input distributions are shifted. In this thesis we attempt to understand how to improve and measure the robustness and generalization properties of these models through a data invariance perspective. Specifically, we investigate whether different types of data invariance can be leveraged during training to improve robustness, as well as whether they can be used after training to characterize and properly calibrate model behavior in different settings. We first propose a method to use denoising autoencoders to generate augmentations along adata manifold, and show that training on these examples improves out-of-domain robustness on a wide range of natural language tasks. Next, we consider a model post-training and show that this same neighborhood invariance can be extended to general transformations and correlates well with model generalization. We apply these principles to the specific problem of blind sequence denoising and demonstrate that our proposed architecture and objective can be used to generate clean DNA sequences from noisy ones without ever observing them during training. Finally, we propose a method to efficiently estimate the invariance of an oracle model’s prediction to arbitrary labellings of a given example, and show how to use these estimates to produce calibrated predictions and measure complexity in both labelled and unlabelled settings.
Source: TSpace (University of Toronto)
Author: Ng, Nathan Hoyen
Institution: University of Toronto
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7132876177