Greedy Byte-Pair Encoding Achieves 0.391 of the Optimum, Past the One-Third Barrier for Packing-Based Analyses
Year: 2026
Type: preprint
Abstract: Byte-Pair Encoding (BPE) builds most language-model tokenisers by repeatedly merging a most frequent adjacent pair of symbols. How much shortening can this greedy rule lose against the best k merges? Kozma and Voderholzer proved that greedy achieves at least one third of the optimal shortening; the best known upper bound is (1-e^{-2})/2 ≈ 0.432. We show that one third is a barrier for packing-based analyses, which see the optimum only through a pair packing, and we prove that greedy achieves at least 0.391 of the optimum on every input, with any tie-breaking. The key idea is rule determinism: all copies of a rule of the optimum spell the same letters, so when one greedy replacement destroys three of the optimum's merge points, half a unit can be refunded on each side. A weaker bound, 0.374, has a short proof that needs no computer; 0.391 adds a certificate in exact arithmetic. We also show that the worst-case ratio is exactly 3/4 for two merges and at most 7/12 for three. Version 2.0 (2026-11-08). Restructured: RC results moved to a companion paper. The re-typing inequality RC and everything proved about it (RC for the first three steps; RC at every step when some optimal merge sequence has tokens of length at most 7, with the corollary that greedy achieves (1-e^{-2})/2 of the best merge sequence with tokens of at most seven letters; the computer-assisted parts and their certificates) are now in the companion paper "Greedy Byte-Pair Encoding Achieves the Tight Constant (1-e^{-2})/2 Against Optima with Short Tokens", doi:10.5281/zenodo.24056290. The Theta(n)-inapproximability of compressed length over unbounded alphabets is now in "Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary Datasets", doi:10.5281/zenodo.24023730. This version keeps the barrier, 0.374, 0.391, rho_2 = 3/4 and rho_3 ≤ 7/12, and is reorganised so that the first twelve pages carry the whole argument: the barrier, the full 0.374 proof, and the 0.391 theorem with its bound stated as one explicit expression (Proposition 11) and a proof outline. New: Section 8 on pair counts with overlaps (as in subword-nmt and Hugging Face tokenizers): when the bounds transfer, a family showing that they do not in general (ratio tending to 1/2 at every budget; 7/16 at budget two with adversarial ties), and a transferred bound of 1/4. The intermediate bound 0.387 is no longer stated (the 0.391 proof never used it). Proofs kept from version 1.2 are unchanged in substance; Lemma 5 now states the invariant (I0) explicitly, the k = 2 case of the main theorem now follows from the warm-up, and the main-theorem appendix renames a few symbols. Previous version: 1.2, doi:10.5281/zenodo.24054238. The record includes the paper, its LaTeX source, and the code and certificates that reproduce every computed result.
Authors: Baliga, Kundan, Xie, Aiden, Byju, Aditya
Language: English
Subfield: Theoretical Computer Science
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225466098