MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
Year: 2024
Type: conference-paper
Abstract: In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. Specifically, previous studies have reported multiple modeling biases that the current tokenization algorithms introduce to non-Latin script languages, the main one being over-segmentation. In this work, we propose MAGNET; multilingual adaptive gradient-based tokenization to reduce over-segmentation via adaptive gradient-based subword tokenization. MAGNET learns to predict segment boundaries between byte tokens in a sequence via sub-modules within the model, which act as internal boundary predictors (tokenizers). Previous gradient-based tokenization methods aimed for uniform compression across sequences by integrating a single boundary predictor during training and optimizing it end-to-end through stochastic reparameterization alongside the next token prediction objective. However, this approach still results in over-segmentation for non-Latin script languages in multilingual settings. In contrast, MAGNET offers a customizable architecture where byte-level sequences are routed through language-script-specific predictors, each optimized for its respective language script. This modularity enforces equitable segmentation granularity across different language scripts compared to previous methods. Through extensive experiments, we demonstrate that in addition to reducing segmentation disparities, MAGNET also enables faster language modelling and improves downstream utility.
DOI: 10.52202/079017-1514
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: Collaborative Research: RI: Small: NL(V)P: Natural Language (Variety) Processing, CAREER: Language Technologies Against the Language of Social Discrimination, NSF-BSF: RI: Small: Efficient Transformers via Formal and Empirical Analysis, NSF-BSF: Collaborative Research: RI: Small: Multilingual Language Generation via Understanding of Code Switching, G6671297155
OpenAlex ID: W4415797757