Work

Pretraining, Quantization, and Evaluation of Romanian Small Language Models

Year: 2026
Type: dissertation
Abstract: This thesis explores the design and evaluation of Small Language Models for the Romanian Language. We build a 21.16 GB curated Romanian corpus from web crawls, targeted scrapers, and literary texts, which translates to approximately 5.6 billion tokens. Four 300M-parameter architectural variants based on different attention mechanisms are pretrained from scratch and evaluated on fluency, sentiment classification, and semantic similarity; these architectures are: Multi-Head Attention, Grouped-Query Attention, Multi-Query Attention, and Sliding Window Attention. Our best model achieves the lowest perplexity on four out of five held-out Romanian domains against existing models pretrained from scratch, including models with up to 10 times more parameters, and all four of our variants surpass every published Romanian baseline on the LaRoSeDa sentiment classification benchmark, including the encoder-based RoBERT-large. We further demonstrate that aggressive 4-bit quantization retains over 98.5% of full-precision fluency, enabling deployment on low-power edge devices under 200 MB.
Institution: Lund University
Language: English
Open Access status: green
OpenAlex ID: W7213388203
Cites: 30
Cited by:
Related to: 10