Improving translation quality with automatic post-editing through large language models
Year: 2024
Type: dissertation
Abstract: With the development of Machine Translation (MT) in the last few years particularly with the emergence of Large Language Models (LLMs) it has been possible to observe the evolution of these systems in terms of translation quality, but also in the development of various tasks that were not possible before neural machine translation (NMT). LLMs are trained with billions of data from the World Wide Web to execute various tasks. Some of the tasks performed by LLMs include text summarization, conversation, grammatical error correction, translation correction and classification, and automatic post-editing (APE), among others. APE relates to methods of learning from data post-edited by a human to improve a machine translation by a system. A system's learning process includes patterns found in parallel data, including an MT version without editing and an MT version with post-editing. APE resurged in 2015 at the Workshop on Machine Translation (WMT) conference, where there were tasks related to APE. Although some models have excelled in their improvements in APE, it was only with the emergence of LLMs that APE resurged in the translation industry. For this reason, the main goal of this project is to improve translation quality with APE through LLMs.The project’s dissertation consisted of experiments with two LLMs: GPT-4 from OpenAI and TowerLLM, Unbabel’s proprietary LLM. The two LLMs were tested using two different content types: customer support and news. The language pair used in our experiments with the customer support data was English to Simplified Chinese (EN_ZH-CN). As a request of the Research Team from Unbabel we also work with the language pair English to Traditional Chinese (EN_ZH-TW) however, we did not make experiments with this data. The language pairs used in our experiments with the news data were English to Simplified Chinese (EN_ZH-CN). However, the data also included the Simplified Chinese to English (ZH-CN_EN) variety. We will not be working with ZH-CN_EN for the customer support data since this language pair does not have a significant volume of data sent by the clients at Unbabel. The customer support content (chat) was from Unbabel proprietary data and the news data was from the Shared Task: General Machine Translation from the WMT23 (Kocmi et al., WMT 2023). The positive results of the experiments of GPT-4 APE and Tower-APE with the customer support data and the Shared Task: General Machine Translation from the WMT23 (Kocmi et al., WMT 2023) data showed that GPT-4 APE performed better than Tower-APE in the two content domains, although Tower-APE presented positive results. The results confirmed the success of the experiments in the two content domains, which show that the use of APE can be applied to diverse content domains. The results also prove that APE can be used through different LLMs such as GPT-4 and TowerLLM. APE could improve the quality of the translation without human intervention. Nevertheless, a human-in-the-loop approach remains important in assessing APE quality. The capacity of GPT-4 APE and Tower-APE to improve the MT is mostly achieved, especially by GPT-4 APE. The introduction of unnecessary errors in their edits is also confirmed. Notwithstanding that there are errors introduced by GPT-4 APE and Tower-APE in both content domains, the number of errors introduced by APE is considerably inferior to the number of correct edits.
Author: Correia, Catarina Cunha do Maio
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7048134618