Year: 2019
Type: conference-paper
Abstract: Segmentation serves as an integral part in many NLP applications including Machine Translation, Parsing, and Information Retrieval.When a model trained on the standard language is applied to dialects, the accuracy drops dramatically.However, there are more lexical items shared by the standard language and dialects than can be found by mere surface word matching.This shared lexicon is obscured by a lot of cliticization, gemination, and character repetition.In this paper, we prove that segmentation and base normalization of dialects can help in domain adaptation by reducing data sparseness.Segmentation will improve a system performance by reducing the number of OOVs, help isolate the differences and allow better utilization of the commonalities.We show that adding a small amount of dialectal segmentation training data reduced OOVs by 5% and remarkably improves POS tagging for dialects by 7.37% f-score, even though no dialect-specific POS training data is included.
DOI: 10.18653/v1/w19-4613
Authors: Mohammed Attia, Ali Elkahky
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W2970822246