Vocabulary Structure and Subword Tokenization Cost in Arabic, Hebrew, and Targumic Aramaic: A Parallel-Text Pilot Study
Year: 2026
Type: preprint
Abstract: This pilot study investigates the vocabulary structure and subword tokenization cost of three related Semitic languages: Arabic, Hebrew, and Targumic Aramaic (specifically, Targum Onkelos on Genesis). Using a parallel corpus of 1,533 verses from the Book of Genesis, the study controls for content differences to isolate the effects of language and orthography on tokenization.We evaluate key vocabulary statistics, including type counts, token counts, type-token ratio (TTR), hapax legomena share, and average word length. Additionally, we measure Byte Pair Encoding (BPE) tokenization fertility across varying vocabulary sizes (250 to 2,000) to assess tokenization efficiency.Our findings reveal a consistent ordering across the three languages: Arabic is lexically the most diverse and costliest to tokenize, Hebrew is the least, and Aramaic sits in between. This gap in tokenization fertility largely reflects the fact that Arabic orthographic words are longer due to attached clitics and rich morphological features. The study underscores the importance of language-specific tokenization strategies and morphological analysis for low-resource NLP in Semitic languages. Future work will extend this corpus to include Syriac (the Peshitta) and additional books, alongside morphological analysis to further test clitic-based explanations.
Source: Figshare
Author: Sareena Bilal
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225465748