Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets
Year: 2023
Type: conference-paper
Abstract: Increasingly larger datasets have become a standard ingredient to advancing the state-of-theart in NLP.However, data quality might have already become the bottleneck to unlock further gains.Given the diversity and the sizes of modern datasets, standard data filtering is not straight-forward to apply, because of the multifacetedness of the harmful data and elusiveness of filtering rules that would generalize across multiple tasks.We study the fitness of task-agnostic self-influence scores of training examples for data cleaning, analyze their efficacy in capturing naturally occurring outliers, and investigate to what extent self-influence based data cleaning can improve downstream performance in machine translation, question answering and text classification, building up on recent approaches to self-influence calculation and automated curriculum learning.
Authors: Irina Bejan, Artem Sokolov, Katja Filippova
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4389519087