Get started
Work

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages

Year: 2023
Type: conference-paper
Abstract: Knowing the language of an input text/audio is a necessary first step for using almost every NLP tool such as taggers, parsers, or translation systems.Language identification is a wellstudied problem, sometimes even considered solved; in reality, due to lack of data and computational challenges, current systems cannot accurately identify most of the world's 7000 languages.To tackle this bottleneck, we first compile a corpus, MCS-350, of 50K multilingual and parallel children's stories in 350+ languages.MCS-350 can serve as a benchmark for language identification of short texts and for 1400+ new translation directions in lowresource Indian and African languages.Second, we propose a novel misprediction-resolution hierarchical model, LIMIT, for language identification that reduces error by 55% (from 0.71 to 0.32) on our compiled children's stories dataset and by 40% (from 0.23 to 0.14) on the FLORES-200 benchmark.Our method can expand language identification coverage into low-resource languages by relying solely on systemic misprediction patterns, bypassing the need to retrain large models from scratch. 1
DOI: 10.18653/v1/2023.emnlp-main.895
Language: English
Open Access status: hybrid
OpenAlex ID: W4389519557
FWCI: 0.533
Cites: 30
Cited by: 3
Related to: 10