Work

Benign Overfitting With Retrieval Augmented Models

Year: 2022

Type: report

Abstract: Despite the fact that modern deep neural networks have the ability to memorize (almost) the entire training set they generalize well to unseen data, contradicting traditional learning theory. This phenomenon --- dubbed benign overfitting --- has been theoretically studied so far in simplified settings only. At the same time, ML practitioners (especially in NLP) figured out how to exploit this feature for more efficient training: retrieval-augmented models (e.g., kNN-LM, RETRO) explicitly store (part of) the training sample in the storage and thus try to (partially) remove a load of memorization from the neural network. In this paper we link these apparently separate threads of research, and propose several possible research directions regarding benign overfitting in retrieval-augmented models.

Source: Nazarbayev University Repository (Nazarbayev University)

Authors: Assylbekov, Zhenisbek, Tezekbayev, Maxat, Nikoulina, Vassilina, Gallé, Matthias

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7199611397

Cites:

Cited by:

Related to: