Online Supplement to: Is Google Gemini better than ChatGPT at evaluating research quality?
Year: 2024
Type: article
Abstract: Purpose: Evaluating the quality of published research for appointments, promotions and departmental assessments is challenging. Whilst recent studies have shown that ChatGPT quality score estimates address some of the limitations of citation-based indicators to support this, it is not known whether other Large Language Models (LLMs) could perform better in this role and this article tests Google Gemini.Design/methodology/approach: Google Gemini 1.5 Flash scores were compared with ChatGPT 4o-mini on evaluations of (a) 51 of the author’s journal articles and (b) up to 200 articles in each of 34 field-based Units of Assessment (UoAs) from the UK Research Excellence Framework (REF) 2021.Findings: From (a), based on correlations with human quality scores, the results suggest that Gemini 1.5 Flash, unlike ChatGPT 4o-mini, may work better when fed with a PDF or article full text, rather than just the title and abstract. From (b), which only evaluated title and abstract inputs, Gemini 1.5 Flash seems to be marginally less able to predict an article’s research quality (using a departmental quality proxy indicator) than ChatGPT 4o-mini, although the differences are small, and both have similar disciplinary variations in this ability. As for ChatGPT, averaging multiple runs of Gemini 1.5 Flash improves the scores.Research limitations: The data is a UK-based sample with UK-centered quality criteria.Originality/value: Overall, Gemini 1.5 Flash seems to be a reasonable but not superior alternative to ChatGPT, while its multimodal focus may have a better ability to work with full text documents.
Source: Figshare
Author: Thelwall, Mike
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6939704716