RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review
Year: 2026
Type: preprint
Abstract: Large language models (LLMs) are increasingly used to evaluate other models’ outputs in medicine, an approach known as LLM-as-a-Judge (LaaJ), yet no LaaJ-specific reporting checklist exists. This scoping review included 91 medical LaaJ studies: 69 from 5,571 database records and 22 from a previous review. Judges were mainly used for response evaluation (74.7%), and 76.9% of the studies used at least one proprietary LLM. Agreement with human evaluators was reported in 65.9% of the studies, with widely varying metrics. Based on these findings, this study developed RULE-LLM, a reporting framework of 21 items and eight sub-items across four domains: role and task specification, underlying data and criteria, LLM-Judge design and output, and evaluation of judge performance. Seven experts rated the initial draft (S-CVI/Ave, 0.94), and five independent raters showed substantial agreement when applying the final framework (Fleiss' κ, 0.67; Gwet's AC1, 0.73). RULE-LLM aims to improve the transparency and reproducibility of LaaJ studies.
Source: Research Square
Authors: Junbok Lee, Huiyeol Yun, Yoon Bin Jung
Institution: Yonsei University
Language: English
Subfield: Health Informatics
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7236793574
