Year: 2026
Type: conference-paper
Abstract: Encoder-only transformers remain essential for practical NLP tasks.While recent advances in multilingual models have improved crosslingual capabilities, low-resource languages such as Latvian remain underrepresented in pretraining corpora, and few monolingual Latvian encoders currently exist.We address this gap by pretraining a suite of Latvian-specific encoders based on RoBERTa, DeBERTaV3, and ModernBERT architectures, including longcontext variants, and evaluating them across a diverse set of Latvian diagnostic and linguistic benchmarks.Our models are competitive with existing monolingual and multilingual encoders while benefiting from recent architectural and efficiency advances.Our best model, lv-deberta-base (111M parameters), achieves the strongest overall performance, outperforming larger multilingual baselines and prior Latvian-specific encoders.We release all pretrained models and evaluation resources to support further research and practical applications in Latvian NLP.
Author: Arturs Znotins
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7140073660