Work

Regularization techniques for fine-tuning in neural machine translation

Year: 2017

Type: conference-paper

Abstract: We investigate techniques for supervised domain adaptation for neural machine translation where an existing model trained on a large out-of-domain dataset is adapted to a small in-domain dataset.In this scenario, overfitting is a major challenge.We investigate a number of techniques to reduce overfitting and improve transfer learning, including regularization techniques such as dropout and L2regularization towards an out-of-domain prior.In addition, we introduce tuneout, a novel regularization technique inspired by dropout.We apply these techniques, alone and in combination, to neural machine translation, obtaining improvements on IWSLT datasets for English→German and English→Russian.We also investigate the amounts of in-domain training data needed for domain adaptation in NMT, and find a logarithmic relationship between the amount of training data and gain in BLEU score.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/d17-1156

Authors: Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, Rico Sennrich

Institution: University of Edinburgh

Language: English

Topic: Domain Adaptation and Few-Shot Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: European Commission

Awards: Translation for Massive Open Online Courses, G434337789, MMT will deliver a language independent commercial online translation service based on a new open-source machine translation distributed architecture