Pair-Merge Tokenisation Is Hard to Approximate on Binary Strings and Strongly NP-Hard on Unary Datasets
Year: 2026
Type: preprint
Abstract: Editorial revision; no result changed. Changes from version 1.1: citations to the authors' companion preprints updated to their current titles and concept DOIs (the byte-pair encoding paper now proves 0.391, cited as such); the shortened "Also in the paper" line under the results box. Supplement unchanged. Tokenisers such as Byte-Pair Encoding (BPE) shorten text by repeatedly merging an adjacent pair of symbols into a new one. We ask whether an efficient algorithm can choose k merges that leave a string of length n almost as short as possible; doing nothing is already within a factor n. We prove that, unless P = NP, no polynomial-time algorithm comes within a factor c n/log n, for some constant c > 0, already for one binary string and for BPE-style merges that replace every occurrence; in the yes case the budget compresses the string to one token. The key idea is that k merges form a grammar with k rules: we turn a vertex-cover gap into a grammar-size gap, amplify it with relabelled copies, encode it in binary, and give explicit BPE-style merges that realise it. For merges that may skip occurrences, a simple algorithm achieves O(n/log log n), and no standard (Karp) NP-hardness proof can go beyond order n/log n unless coNP is contained in NP/poly. Over a one-letter alphabet, both tokenisation problems left open by Kastreva et al. are strongly NP-complete on datasets with repeated strings. Files: the paper (PDF), the arXiv-style LaTeX source, and a supplementary archive with all verification scripts and their results (MIT licence). Version 1.1 changes (editorial only; every theorem, lemma and proof is unchanged): shorter abstract; introduction with a three-item results box, a key-idea figure for the bridge between merges and grammars, and a short glossary of complexity terms; the warm-up over a large alphabet is made self-contained, with a figure for the witness argument and a paragraph listing what can be checked by hand on a single edge; the paper is organised in three parts (hardness; algorithms and the limits of hardness proofs; discussion); case checks of the binary full-merge schedule, the parameter choices of the main theorem and the dataset corollary moved to a new appendix; new figures for the unary full-merge structure and for the approximability landscape and the Karp barrier; a connections paragraph relating constant-factor guarantees for tokens saved (BPE, EvenOdd, GreedTok) to the hardness for tokens left; one scope and limitations section. Theorems, lemmas and appendices from Section 5 onward are renumbered; the supplement README gives the mapping.
Authors: Baliga, Kundan, Xie, Aiden, Byju, Aditya
Language: English
Subfield: Theoretical Computer Science
Field: Mathematics
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225400471