Work

Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation?

Year: 2023

Type: conference-paper

Abstract: Pre-training masked language models (MLMs) with artificial data has been proven beneficial for several natural language processing tasks such as natural language understanding and summarization; however, it has been less explored for neural machine translation (NMT).A previous study revealed the benefit of transfer learning for NMT in a limited setup, which differs from MLM.In this study, we prepared two kinds of artificial data and compared the translation performance of NMT when pretrained with MLM.In addition to the random sequences, we created artificial data mimicking token frequency information from the real world.Our results showed that pre-training the models with artificial data by MLM improves translation performance in low-resource situations.Additionally, we found that pre-training on artificial data created considering token frequency information facilitates improved performance.

Source: Findings of the Association for Computational Linguistics: EACL

DOI: 10.18653/v1/2023.findings-eacl.166

Authors: Hiroto Tamura, Tosho Hirasawa, Hwichan Kim, Mamoru Komachi

Institution: Tokyo Metropolitan University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W4386566471

FWCI: 0.111

Cites: 21

Cited by: 1

Related to: 10