Year: 2025
Type: dissertation
Abstract: Sentence embeddings are a core part of many modern Natural Language Processing ap plications, such as search engines. Their multilingual variants, which are able to produce vector representations for many languages, exhibit significantly lower performance in underrepresented languages compared to English. Also, the dimensionality of the pro duced embeddings and the model’s size might be prohibitive in terms of memory and cost. In this work, we develop two sentence embedding models using the Sentence-BERT (SBERT) framework for the Portuguese language, employing the Matryoshka Represen tation Learning (MRL) loss function. The goal is to enhance vector representations for the Portuguese language while maintaining a reduced dimensionality with no computa tional overhead. We assess the performance of the proposed models against their variants trained without the MRL approach and against other open-source Portuguese and multi lingual models. Through this evaluation method, we found that MRL-based models (i) perform better at lower dimensionality than their non-MRL counterparts by 0.01-0.02 points in the Semantic Textual Similarity (STS) benchmark using the Pearson Correlation metric; (ii) have better performance in the STS benchmark than the multilingual models, with at least 0.06 points; and (iii) present competitive performance against state-of-the art Portuguese embedding models, while being trained on much less data. Additionally, we also evaluate the proposed model’s performance when being used with float32 and bfloat16 representations, with little to no performance variation while occupying 50% less memory.
Author: Pinto, Gabriel Vogel
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6992952617