Rule-Orbit Vocabulary (ROV)
Year: 2026
Type: preprint
Abstract: Rule-Orbit Vocabulary (ROV): Lossless Vocabulary Factorization for Subword Tokenizers via Automatically Mined Rewrite Rules This record contains a preprint introducing Rule-Orbit Vocabulary (ROV), a method for reducing the embedding and softmax size of language models without changing tokenization. OverviewByte-pair-encoding (BPE) vocabularies devote many entries to surface variants of the same word that differ only by a simple rewrite, such as a leading space, capitalization or a common affix (e.g., walk, walks, walked, walking). Each variant needs its own embedding row and output logit, and rare variants are under-trained. Conventional vocabulary pruning removes such entries but lengthens token sequences. ROV takes a different approach. It keeps the original segmentation intact and re-identifies each token as a (base, rule) pair. Method Rewrite rules (prefix and suffix add/strip) are mined automatically from an existing tokenizer vocabulary. A distributional similarity gate admits a rule instance only when the paired tokens appear in similar contexts. Tokens are grouped into "orbits" around a canonical base form and encoded losslessly as (base_id, rule_id). The model uses an additive input embedding and a factorized output head, p(b | h) · p(r | b, h). Sequence length and bytes-per-token are unchanged by construction. Key findings In parameter-matched experiments on GPT-2-tokenized text across three small transformer sizes (128×2, 256×4, 512×8), case-based folding reduced loss in all 14 completed runs, by about 0.5–1.2% of total loss. The largest and most consistent gains were on rare variant tokens. These gains are partly confounded with a reallocation of parameters to wider feed-forward layers. Retrained 44k-entry tokenizers outperformed ROV by 0.025–0.044 bits per byte in a separate comparison. The paper reports these results transparently, positions ROV relative to concurrent work on compositional and case-aware tokenization, and outlines the controlled experiments needed to isolate the effect of folding. Contents ROV_Research_Paper.docx: full preprint ROV_Abstract.docx: one-page abstract StatusWorking paper and preprint. The results are preliminary, and further experiments are in progress.
Author: Ujjwalkumar Bharatkumar Soni
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226017338
