Work

Comparative Analysis of Sentence Transformers for Reference Paper Collection in Five Academic Fields

Year: 2025
Type: conference-paper
Abstract: Nowadays, general-purpose literature search engines like Google Scholar are commonly used to find reference papers by scholars. Although they perform well in a particular discipline, users may face difficulties maintaining relevance in others. The limitation in handling queries containing domain-specific vocabulary, struggles in capturing the topic’s semantic context, and the matching techniques present challenges. Currently, we are developing a reference paper collection system to accelerate academic reference searches through integrating multiple database API, semantic sentence embedding models, and web scraping technology. However, our prior evaluations were limited in using predefined queries under controlled environments and relying on lightweight embedding models. In this paper, we conduct a comparative analysis of sentence transformer models including all-MiniLM-L6-v1, all-DistilRoBERTa-v1, all-mpnet-base-v2, SciBERT, and SPECTER across five academic fields with user-defined queries. We adopt precision@10, NDCG@10, accessibility@10, and response time for evaluation metrics. The results show that some models excel in all academic fields, while others perform well in domain-specific contexts, highlighting the optimal trade-off between relevance, semantic domain understanding, and time efficiency in academic reference paper collection systems. Significant differences in performance metrics across models were confirmed statistically in our experiments.
Language: English
Open Access status: hybrid
OpenAlex ID: W7127064384
FWCI: 1.441
Cites: 13
Cited by: 4
Related to: 10