Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese
Year: 2022
Type: conference-paper
Abstract: Multilingual language models such as mBERT have seen impressive cross-lingual transfer to a variety of languages, but many languages remain excluded from these models.In this paper, we analyse the effect of pre-training with monolingual data for a low-resource language that is not included in mBERT -Maltese -with a range of pre-training set ups.We conduct evaluations with the newly pre-trained models on three morphosyntactic tasks -dependency parsing, part-of-speech tagging, and named-entity recognition -and one semantic classification task -sentiment analysis.We also present a newly created corpus for Maltese, and determine the effect that the pre-training data size and domain have on the downstream performance.Our results show that using a mixture of pre-training domains is often superior to using Wikipedia text only.We also find that a fraction of this corpus is enough to make significant leaps in performance over Wikipedia-trained models.We pre-train and compare two models on the new corpus: a monolingual BERT model trained from scratch (BERTu), and a further pretrained multilingual BERT (mBERTu).The models achieve state-of-the-art performance on these tasks, despite the new corpus being considerably smaller than typically used corpora for high-resourced languages.On average, BERTu outperforms or performs competitively with mBERTu, and the largest gains are observed for higher-level tasks.
Institutions: University of Malta, Utrecht University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: Università ta' Malta
OpenAlex ID: W4281492541