Year: 2023
Type: conference-paper
Abstract: Recent work attributes progress in NLP to large language models (LMs) with increased model size and large quantities of pretraining data.Despite this, current state-of-the-art LMs for Hebrew are both under-parameterized and under-trained compared to LMs in other languages.Additionally, previous work on pretrained Hebrew LMs focused on encoderonly models.While the encoder-only architecture is beneficial for classification tasks, it does not cater well for sub-word prediction tasks, such as Named Entity Recognition, when considering the morphologically rich nature of Hebrew.In this paper we argue that sequence-to-sequence generative architectures are more suitable for large LMs in morphologically rich languages (MRLs) such as Hebrew.We demonstrate this by casting tasks in the Hebrew NLP pipeline as text-to-text tasks, for which we can leverage powerful multilingual, pretrained sequence-to-sequence models as mT5, eliminating the need for a separate, specialized, morpheme-based, decoder.Using this approach, our experiments show substantial improvements over previously published results on all existing Hebrew NLP benchmarks.These results suggest that multilingual sequence-to-sequence models present a promising building block for NLP for MRLs.
Institution: Google (United States)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Funder: Bar-Ilan University
OpenAlex ID: W4385572234