Year: 2025
Type: conference-paper
Abstract: Assessing a research paper's scholarly impact is an important phase in the scientific research process; however, metrics typically take some time after publication to accurately capture the impact.Our study examines how Large Language Models (LLMs) can predict scholarly impact accurately.We utilize Retrieval-Augmented Generation (RAG) to examine the degree to which the LLM performance improves compared to zero-shot prompting.Results show that LLama3-8b with RAG achieved the best overall performance, while Gemma-7b benefited the most from RAG, exhibiting the most significant reduction in Mean Absolute Error (MAE).Our findings suggest that retrieval-augmented LLMs offer a promising approach for early research evaluation.Our code and dataset for this project are publicly available 1 2 .
Institutions: University of North Texas, Northern Illinois University
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4412944121