Work

OdiaSLM: From-Scratch Pretraining, Morphology-Aware Tokenization, and Calibrated Evaluation of 60M and 100M Small Language Models for Odia

Year: 2026
Type: preprint
Abstract: While frontier multilingual Large Language Models (LLMs) impose a severe tokenization tax on morphologically rich, low-resource Indic scripts such as Odia (ory_Orya)—fragmenting words into 5-6x more subword tokens than English and inflating inference latency—compact, script-native models remain underexplored. We present OdiaSLM-60M (62.11M parameters) and OdiaSLM-100M (100.69M parameters), two sovereign decoder-only Small Language Models pre-trained from scratch for Odia on up to 1.57 billion tokens drawn from a strictly deduplicated and 13-gram benchmark-decontaminated corpus of 9.62 million documents (1.247 billion words). First, we design a 16,384-vocabulary canonical NFC-normalized SentencePiece BPE tokenizer that preserves Odia juktakshara conjuncts, achieving a fertility of 1.4809 tokens/word (11.57 UTF-8 bytes/token), 0 <unk> tokens, and a 3.6x compression advantage over general multilingual tokenizers. Second, utilizing Grouped-Query Attention (GQA), SwiGLU, Half-Split RoPE, FP32-variance RMSNorm, auxiliary Z-loss stabilization, a Warmup-Stable-Decay (WSD) schedule, and two-stage context extension (512 to 1024), we demonstrate clean neural scaling from 62.11M (validation perplexity 19.92, Bits-Per-Byte 0.3965) to 100.69M parameters (validation perplexity 17.14, Bits-Per-Byte 0.3767, Top-1 token accuracy 44.61%). Third, we conduct a comprehensive zero-shot evaluation across 27 tasks in the Odia Evaluation Benchmark, uncovering both cross-lingual untranslated choice artifacts in 9 tasks and severe surface-form competition under raw log-likelihood. Applying Pointwise Mutual Information (PMI) calibration unlocks a macro-average accuracy of 60.83% across the 18 pure-Odia benchmarks (+34.44% lift over random chance). Both models are released in native LlamaForCausalLM format.
Author: Misra, Arun
Language: English
Open Access status: green
OpenAlex ID: W7225903371
Cites:
Cited by:
Related to: 10