Evaluating Embedding Models: Investigating Factors Affecting Performance Through the Medical Domain
Year: 2025
Type: article
Abstract: In this paper, we investigated the performance of embedding models on two different domains, using models of different embedding dimensions, and even varying dimension to fully understand the impacting factors for the models. We ran tests for seven state-of-the-art models across natural questions and medial QA datasets to investigate the models' out-of-domain performance, and how factors such as the number of training parameters and fine-tuning affect accuracy and speed. We also explored the potential impact of several other variables, such as how model architecture could be the only factor affecting processing speed and how training data could have a significant effect on out-of-domain accuracy. The results show that the accuracy of the models follows a decreasing exponential relationship with the number of embedding dimensions and that the number of training parameters has little effect on model performance. The models we found to have the best accuracy were the nq-distillbert for natural questions and MedEmbed-base for medical QA. The model with the lowest encode time and the best accuracy-to-speed ratio by far across both datasets, as well as the best out-of-domain performance, was the all-miniLM.
Author: Eson Pu
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4415133194
FWCI:
Cites:
Cited by:
Related to: