Estimating Cross-Lingual Tokenization Fragmentation Risk: A Frequency-Based Uniqueness Score for Geographic Entities
Year: 2026
Type: preprint
Abstract: Large language models often penalize multilingual users through hidden 'tokenizer taxes' caused by subword fragmentation. Geographic proper nouns offer a controlled stress case for this distributional asymmetry, as they are standardized, cross-linguistically available entities whose orthographic forms frequently mismatch tokenizer training distributions. We introduce the Cross-Linguistic Orthographic Uniqueness Score (CLOUS), a character-level metric based on Shannon surprisal that quantifies the orthographic rarity of localized country names. We evaluate CLOUS on a controlled corpus of 249 ISO-registered country names; generalization to other geographic entity types remains future work. We evaluate CLOUS against tokenization fertility for six production tokenizers spanning three algorithm families across eleven languages and seven script families. Using Spearman rank correlation with Benjamini-Hochberg correction, CLOUS shows significant associative strength with fertility in 61 of 65 valid language-tokenizer combinations. As a screening tool for high-fragmentation entities, CLOUS achieves ROC-AUC = 0.850 and F1 = 0.740, correctly identifying approximately three-quarters of the highest-fragmentation entities from orthographic properties alone. We further show that CLOUS correlates with fertility more strongly than length-based baselines across all evaluated scripts, and outperforms a character bigram surprisal model in 69.2% of tested combinations, particularly for logographic, syllabic, and Latin scripts. Conversely, the bigram baseline proves superior for morphophonologically complex scripts like Arabic and Devanagari, indicating that tokenizer fragmentation is driven by local orthographic context in some scripts and absolute character rarity in others. Script family appears to be the primary driver of CLOUS-fertility correlation strength, with effect sizes ranging from modest for Latin-script languages ) to large for non-Latin scripts ( up to 0.686 for Japanese). These findings suggest that CLOUS may be useful as a lightweight pre-tokenization diagnostic for multilingual NLP pipelines.
Author: Rivera, Richard
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222890746
Cites:
Cited by:
Related to: