Effectively compiling parallel corpora for machine translation in resource-scarce conditions
Year: 2023
Type: dissertation
Abstract: For machine translation (MT) systems to produce accurate and fluent translations, reliable parallel corpora are key. \nErrors, due to misalignments or inadequate filtering during compilation of a parallel corpus, can have detrimental effects on the performance of an MT system trained on the data. Moreover, when the corpus is too small, the MT system may not be able to capture the complexities of the source and target languages and produce accurate translations. However, obtaining high-quality parallel data is often a challenging task, even more so for languages with a low number of speakers or rich morphology exacerbating the data sparsity problem. It is thus imperative to develop accurate methods for processing parallel corpora that can help make the most of what is available. \n \nIn this thesis, we address this challenge by exploring various methods for processing parallel corpora to maximize their usefulness for MT. First, we investigate a variety of classifiers and scoring mechanisms used for filtering parallel corpora, looking into how efficient they are at removing data detrimental to MT training and retaining useful data. We find that different filtering strategies suit different datasets and that filtering separately for different translation directions can yield better translations in downstream MT tasks. Second, we examine different approaches to sentence alignment, compare their effectiveness, and show that combining multiple methods can improve alignment accuracy. Third, we experiment with comparable corpora mining methods to extract even more useful data from sentences that had previously been discarded, showing that this often overlooked data is a potential source of useful training data. Finally, we manually evaluate translations generated by MT systems trained on our processed datasets, most suitable for each translation direction, confirming the advantages of our applied methods. \n \nOur findings highlight the importance of careful processing and curation of parallel corpora for MT. We propose approaches for maximizing the utility of available parallel data, particularly for scenarios where resources are scarce, contributing to the development of more accurate and reliable MT systems.
Source: Opin vísindi (Opin vísindi)
Author: Steingrímsson, Steinþór
Institution: University of Alicante
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7046929794
