Get started
Work

Proportionate in Volume, Distinctive in Composition: Persian in the Open Model and Dataset Ecosystem of Hugging Face

Year: 2026
Type: preprint
Abstract: Purpose. Open model hubs have become the main infrastructure through which language technologies are shared, yet there is no quantitative audit of how well Persian is served relative to comparable languages. We ask whether Persian resources are scarce relative to the language's digital footprint, how concentrated their production and use are, and how their composition differs from peers. Design and methods. We took a full census of all models and datasets on Hugging Face tagged with each of 20 languages (Persian and 19 comparison languages; 5,052 Persian-tagged models and 1,452 datasets), retrieved on 1 October 2026 through the public API, and combined it with Wikipedia size as a measure of digital footprint. We separate language-dedicated resources (a single language tag) from multilingual ones, estimate how many dedicated models a language's footprint predicts using log–log regression on the other 19 languages (with prediction intervals), and compare concentration of authors and downloads, task mix, licensing, base-model dependence and creation timing. Findings. Persian had 913 dedicated models (18.1% of Persian-tagged models) and 569 dedicated datasets. Relative to the footprint-based prediction (1,593 models) Persian was at 0.57 of the expected volume (95% prediction interval 0.13–2.53), and ratios under alternative footprint measures and definitions ranged from 0.42 to 0.75, always with intervals that include parity; the data therefore do not show a statistically distinguishable shortfall. Production and use were more concentrated than for peers: five authors made 29.6% of the dedicated models (peer median 19.3%) and three authors accounted for 94.5% of dedicated-model downloads (peer median 77.8%). Multilingual models held 81.7% of Persian-tagged models and 99.3% of downloads, which is typical (peer medians 86.8% and 98.5%). Composition was distinctive: speech recognition models were 29.6% of dedicated models (peers 14.7%), question answering 8.4% (peers 1.7%), and text generation 12.4% (peers 21.8%), and generative models were the 17th of 20 languages in share. Fewer dedicated Persian models than peers were created since 2023 (85.2% versus 89.6%), and the share of text generation among them was 13.5% versus 18.6%. Originality. This is, to our knowledge, the first language-specific quantitative audit of Persian on a major model hub; it shows that the practical problem is concentration, generative under-provision and dependence on a few maintainers rather than a raw shortage of artefacts, and it documents measurement pitfalls of hub metadata.
DOI: 10.5281/zenodo.23448550
Language: English
Open Access status: green
OpenAlex ID: W7222901733
Cites:
Cited by:
Related to: