Year: 2026
Type: conference-paper
Abstract: Multilingual NLP often relies on dataset counts from centralized catalogues to characterize which languages are resource-rich or resource-poor. However, these catalogues record only one layer of dataset visibility: what has been registered or institutionally distributed. They do not necessarily reflect which datasets are created, cited, or reused in the research literature. To examine this gap, we combine a catalogue-based baseline with literature-backed evidence of dataset circulation. We introduce the Resource Density Index (RDI), defined as the number of catalogued datasets per one million speakers, and compute it for the 200 most widely spoken languages in Ethnologue. Among them, 118 languages (59%) have an average RDI of zero across the LRE Map and the Linguistic Data Consortium (LDC), and another 23 fall below 0.1, corresponding to at most one catalogued dataset per ten million speakers. We then apply an LLM-assisted citation-mining pipeline over the Semantic Scholar corpus to these 141 low-visibility languages. After manual validation and consolidation, we identify 609 unique datasets across 53 languages, of which 356 remain openly accessible through working public links. These results reveal a substantial visibility gap: many large-speaker languages appear data-poor in catalogue records yet show clear evidence of dataset activity in the research literature. Our findings suggest that multilingual data scarcity should be understood not only as a production problem, but also as a question of documentation, discoverability, and long-term accessibility. Code and data are publicly available at (https://github.com/zhiyintan/dataset-visibility-asymmetry).
Authors: Zhiyin Tan, Changxu Duan
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Reduced inequalities
Open Access status: green
Awards: Verbundprojekt: InsightsNet - Entwicklung einer Meta-Methodik und eines konzeptuellen Rahmens zur transdisziplinären Tiefenerschließung und Analyse multimodaler digitaler Objekte. Demonstriert an den Use Cases KI- und Klimawandel-Diskurse Teilprojekt: Analyse und Bereitstellung vernetzter multimodaler Korpora, hessianAISC - hessianAI Service Center
OpenAlex ID: W7160016883
FWCI:
Cites:
Cited by:
Related to: