Parameter-efficient fine-tuning of large language models for Tatar language: a comparative study of LoRA and QLoRA
Year: 2026
Type: conference-paper
Abstract: This article investigates the application of large language models (LLMs) to the Tatar language. We evaluate parameter-efficient fine-tuning methods, LoRA and QLoRA, under limited computational resources using a subset of the IPSAN/tat_monocorpus_v2 corpus ($\mathbf{1 0, 0 0 0}$ samples). Fully finetuned GPT-2 family models ($\mathbf{1 2 4 M - 3 5 4 M}$ parameters) serve as baselines, alongside state-of-the-art models Mistral, Qwen, and DeepSeek adapted via QLoRA. Our results show that QLoRA adaptation of Mistral-7B achieves a perplexity of 3.98 on the test set, comparable to fully fine-tuned gpt2-medium (2.91). Notably, QLoRA requires training only $\mathbf{4 1. 9 M}$ parameters ($\boldsymbol{\lt} \mathbf{0. 6 \%}$ of the full model), two orders of magnitude fewer than full fine-tuning. Mistral-7B outperforms Qwen2.5-7B (PPL 5.44) and DeepSeek-7B (PPL 5.32) in adapting to Tatar. Increasing LoRA rank from 8 to 16 consistently improves performance across all architectures. Qualitative analysis confirms that Mistral-7B with QLoRA generates coherent, grammatically correct Tatar texts with accurate usage of language-specific characters (ә, ө, ң) and agglutinative forms. These findings demonstrate that parameter-efficient adaptation of multilingual LLMs is a promising approach for developing language models for Tatar and other Turkic languages.
Institutions: Kazan Federal University, Tatarstan Academy of Sciences
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7167262968