Revisiting Graphones for Language Modelling
Year: 2026
Type: preprint
Abstract: Conventionally, text-based vector-symbolic codes have used letters or bytes as their atomic units. We propose using the graphone instead: a unit we adapt from grapheme-to-phoneme conversion that pairs a letter with its corresponding sound. We find that graphones outperform letters and bytes as they let the model utilise the individual streams to build a signal stronger than its constituent parts. We first benchmark graphones against bytes and letters on prediction, grammatical judgement and part-of-speech (POS) selectivity, finding reductions of 0.034 to 0.037 bits per byte that remain significant after Holm correction. We then investigate whether these gains come from coupling each letter and its sound into a single unit or from providing them as separate streams, finding the separate streams to be sufficient: the model keeps weight on the sound stream even when the joint unit is present; without the joint unit, the gains remain.
Author: Chetham, Ruadhan
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7236623959
