Work

SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

Year: 2020

Type: conference-paper

Abstract: Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples.Data augmentation is a common method used to prevent overfitting and improve OOD generalization.However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold.We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold.We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models.In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English. 1

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2020.emnlp-main.97

Authors: Nathan Ng, Kyunghyun Cho, Marzyeh Ghassemi

Institutions: University of Toronto, Vector Institute, Courant Institute of Mathematical Sciences, New York University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid