Envisioning the Future of Peer Review: Investigating LLM-Assisted Reviewing Using ChatGPT as a Case Study
Year: 2025
Type: conference-paper
Abstract: Peer review ensures research quality but faces growing challenges. Large Language Models (LLMs) offer potential solutions, yet most research has focused on independent AI-generated reviews rather than human-AI collaboration. We conducted a within-subject experiment with 24 HCI reviewers to evaluate LLM-assisted peer review’s impact on review efficiency, quality, workload, and user perceptions. Analysing subjective ratings, objective completion data, qualitative interviews, and chat history, we found that ChatGPT significantly reduces workload but does not substantially shorten review time or improve review quality. Reviewers valued its support for summarisation, information retrieval, idea generation, and confidence-building but noted challenges in content verification, interaction inefficiencies, and the need for human oversight. Our findings contribute to the broader conversation on the role of technology in scholarly publishing, offering design insights for supportive review systems that complement rather than replace human expertise, and highlighting the importance of responsible integration in academic peer review.
Authors: Shiping Chen, Duncan Brumby, Anna Cox
Institution: University College London
Language: English
Subfield: Library and Information Sciences
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W4411505952