Year: 2023
Type: conference-paper
Abstract: Language identification (LID) is a fundamental step in many natural language processing pipelines.However, current LID systems are far from perfect, particularly on lower-resource languages.We present a LID model which achieves a macro-average F1 score of 0.93 and a false positive rate of 0.033% across 201 languages, outperforming previous work.We achieve this by training on a curated dataset of monolingual data, the reliability of which we ensure by auditing a sample from each source and each language manually.We make both the model and the dataset available to the research community.Finally, we carry out detailed analysis into our model's performance, both in comparison to existing open models and by language class.
Institution: University of Edinburgh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4385572245