Year: 2026
Type: conference-paper
Abstract: Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage.Widelyused subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio.To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers.IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-percharacter distribution across languages.We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.Github github.com/Mikki99/ipa-tokenization
Institutions: University of Amsterdam, Stanford University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: European Commission
OpenAlex ID: W7166818500