Get started
Work

Comparative Analysis of Embedding Models for Matching Questions and Contexts in the Kazakh Language

Year: 2025
Type: article
Abstract: This paper presents a comparative analysis of the effectiveness of modern embedding models for matching questions and contexts in the Kazakh language, within the framework of semantic search and question-answering (QA) systems. The study is based on the open dataset Kundyzka/informatics_kaz, which contains approximately 7,700 question–context pairs covering key concepts in computer science. The analysis focuses on comparing the traditional TF-IDF method with state-of-the-art multilingual models, including mMiniLM, LaBSE, Alibaba GTE-multilingual-base, intfloat/multilingual-e5-small, and Snowflake Arctic-Embed v2.0. The evaluation metrics used are Accuracy@1, Mean Reciprocal Rank (MRR), and ROC AUC. The zero-shot experiments demonstrate that embedding-based models significantly outperform TF-IDF, achieving Accuracy@1 in the range of 0.59–0.63, whereas the baseline method yields 0.50. Moreover, visualization results using t-SNE indicate that modern models are able to cluster semantically related questions and contexts more closely, which positively affects the quality of ranking. Among the models considered, Alibaba GTE and Snowflake Arctic-Embed v2.0 achieved the best results, confirming their relevance for developing automatic question-answering systems in educational AI applications. This study highlights that even without fine-tuning on Kazakh data, pretrained multilingual models can substantially enhance semantic matching quality, offering promising prospects for their use in educational systems and other information services.
Source: КазУТБ
DOI: 10.58805/kazutb.v.3.28-987
Authors: K. Mazhitova, A. Tleubayeva, S. Mukhammediya, A. Tanirbergenova, A. Toktaganova, Zh. Kambarova
Language: English
Open Access status: hybrid
OpenAlex ID: W7084590793
FWCI:
Cites:
Cited by:
Related to: