HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization
Year: 2025
Type: conference-paper
Abstract: Many pre-trained language models (PLMs) exhibit suboptimal performance on mid-and lowresource languages, largely due to limited exposure to these languages during pre-training.A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.Among such methods, OFA (Liu et al., 2024a) proposes a similarity-based subword embedding initialization heuristic that is both effective and efficient.However, OFA restricts target-language token embeddings to be convex combinations of a fixed number of source-language embeddings, which may limit expressiveness.To overcome this limitation, we propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding initialization.The hypernetwork is trained to map from an external multilingual word vector space to the PLM's token embedding space using source-language tokens. 1 Once trained, it can generate flexible embeddings for target-language tokens, serving as a good starting point for continual pretraining.Experiments demonstrate that HYPEROFA consistently outperforms random initialization baseline and matches or exceeds the performance of OFA in both continual pre-training convergence and downstream task performance.We make the code publicly available.2
Authors: Enes Özeren, Yihong Liu, Hinrich Schuetze
Institution: Ludwig-Maximilians-Universität München
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412889452