Get started
Work

The Orthographic and Encoding Convention Tax: Measuring Systematic Tokenization Cost Asymmetries in Meaning-Preserving English Formatting Choices

Year: 2026
Type: preprint
Abstract: Language model APIs charge per token, and tokenizer vocabularies reflect whatever text happened to be common in their training corpora rather than any principled notion of formatting equivalence. However, whether this creates a systematic cost asymmetry across everyday orthographic and encoding choices within a single language remains unmeasured. This gap matters because writers, API users, and training-data curators make hundreds of small formatting decisions daily, each with an invisible and potentially unequal token price. To address this gap, we test 19 categories of meaning-preserving formatting variation, including smart versus straight punctuation, dash styles, Unicode normalization form, emoji composition, and six others, across 45 tokenizers spanning 30 independent tokenizer families. Compared with prior tokenizer-fairness work, our study offers three advantages: it isolates formatting choice from content and language, it applies a uniform statistical protocol across every category, and it includes two direct mechanism probes that convert correlational patterns into architectural explanations. We evaluate the tax on 2,828 hand-templated minimal pairs, producing 118,776 tokenizer-pair encodings. Our experiments show that five categories carry a completely or near-completely unanimous direction across all 30 tokenizer families, that a tokenizer's measured Unicode-normalization behavior explains the entire direction of two categories with near-certainty (p < 10⁻¹¹), and that which stylistic category a pair belongs to explains roughly nine times more variance in token cost than which tokenization algorithm was used. More importantly, we find that this asymmetry is not evenly distributed across formatting conventions: contractions of a sentence into a compact word, a dropped Oxford comma, or a trailing abbreviation period each carry a reliable, unanimous cost that a writer pays without knowing it. This pattern is attributable to a simple observation: tokenizer vocabularies are learned artifacts, and a formatting convention's token cost depends on how often that exact byte sequence appeared during vocabulary training, not on how much information the convention conveys. This result suggests that token-based billing quietly rewards some formatting registers over others, independent of content, in the same way that cross-lingual tokenizer unfairness quietly rewards some languages over others.
Language: English
Open Access status: green
OpenAlex ID: W7170063457
Cites:
Cited by:
Related to: