Not Just Under-Resourced, but Under-Represented: A Linguistic Analysis of Transformer Failures on Austroasiatic Morphosyntax
Year: 2025
Type: preprint
Abstract: Large multilingual language models have become a cornerstone of modern NLP, yet their success is largely confined to a typologically homogeneous set of high-resource languages [4]. This paper argues that the challenges faced by low-resource languages are not merely a function of data scarcity but of profound typological divergence. We present a multi-layered analysis of model failures on Khasi, an Austroasiatic language of Northeast India whose Verb-Subject-Object (VSO) word order and isolating morphology are structurally distant from the Indo-Aryan and Dravidian languages that dominate regional and global models. Using a high-performing dedicated model, KhasiBERT (Nyalang, in prep.), as a linguistic probe, we systematically dissect the failures of multilingual models like mBERT, XLM-R, and the region-focused MuRIL. Our analysis reveals that these failures begin at the input layer, with multilingual tokenizers catastrophically fragmenting Khasi words into meaningless sub-units. This leads to a collapse in semantic prediction and, most revealingly, a deceptive performance in few-shot learning, where models overfit to surface-level artifacts of their own flawed tokenization rather than learning true grammatical structure. Our findings provide compelling empirical evidence that achieving genuine linguistic inclusivity in AI requires a shift from simply adding more languages to developing models that are fundamentally aware of typological diversity.
Author: Badal Nyalang
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: gold
OpenAlex ID: W4414364837
Cites:
Cited by:
Related to: