Year: 2025
Type: article
Abstract: Peer review is a cornerstone of science, ensuring the integrity of scholarly communication. However, it has faced criticism for potential bias and harsh language style. This study explores the language style of peer reviews written by large language models (LLMs), specifically GPT-4, to examine whether it can mitigate such issues. By comparing human reviews from NIPS 2022 with GPT-4-generated reviews under different tonal conditions (normal, kind, harsh), this pilot study assesses the objectivity, clarity, and sentiment of AI-generated feedback. Using sentiment and subjectivity analyses, we find that GPT-4’s reviews exhibit lower subjectivity and more positive sentiment, even under prompts encouraging harshness. GPT-4 also avoids outright rejection and offers more favorable outcomes. However, this tendency to minimize negative feedback raises concerns about its critical evaluation capacity. The results suggest that LLMs may complement human peer review but must be used cautiously to maintain rigor. Future work will expand this analysis using a large-scale dataset to further validate these findings.
Authors: Zheng, Xiang, Xiong, Tiancheng
Institution: University of Wisconsin–Madison
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220826385