Using Morphological Lexicons for Weakly Supervised Subword Vocabulary Learning and Bias-Aware Evaluation
Year: 2026
Type: preprint
Abstract: Subword vocabularies are usually learned from corpus statistics. This is useful for coverage, but it can produce units that cross morpheme boundaries and make the resulting vocabulary less transparent. This article studies a simple use of a morphological lexicon in subword vocabulary learning. We convert MorphoLex into prefix, root/stem, and suffix inventories, and use these inventories as weak supervision during the training of BPE, WordPiece-style, and Unigram tokenizers. The change is limited to vocabulary learning: the learned tokenizer is applied with the standard inference procedure of the base method. The same resource also motivates a bias-aware evaluation protocol, with controls for affix-completeness bias and for word-level provenance overlap between the lexicon used for guidance and the words used for evaluation. On MorphoLex-derived English test sets, lexicon-guided training improves morpheme-boundary F1 by +5.00 pp for BPE, +3.23 pp for Unigram, and +1.10 pp for WordPiece-style training at the best tested settings. The gains persist when the affix-completeness constraint is removed and when guidance and evaluation use disjoint MorphoLex word pools. The results support a modest but useful claim: morphological lexicons can serve both as weak supervision for tokenizer training and as a resource for controlled evaluation.
Source: Research Square
Authors: Yifei Li, Zijun Li, Hui Xiao, Xuefeng Li
Institution: Tongji University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225733179