Work

RULE-LLM reporting framework for large language model-as-a-judge evaluation in medicine developed from a scoping review

Year: 2026
Type: preprint
Abstract: Large language models (LLMs) are increasingly used to evaluate other models’ outputs in medicine, an approach known as LLM-as-a-Judge (LaaJ), yet no LaaJ-specific reporting checklist exists. This scoping review included 91 medical LaaJ studies: 69 from 5,571 database records and 22 from a previous review. Judges were mainly used for response evaluation (74.7%), and 76.9% of the studies used at least one proprietary LLM. Agreement with human evaluators was reported in 65.9% of the studies, with widely varying metrics. Based on these findings, this study developed RULE-LLM, a reporting framework of 21 items and eight sub-items across four domains: role and task specification, underlying data and criteria, LLM-Judge design and output, and evaluation of judge performance. Seven experts rated the initial draft (S-CVI/Ave, 0.94), and five independent raters showed substantial agreement when applying the final framework (Fleiss' κ, 0.67; Gwet's AC1, 0.73). RULE-LLM aims to improve the transparency and reproducibility of LaaJ studies.
Institution: Yonsei University
Language: English
Field: Medicine
Open Access status: green
OpenAlex ID: W7236793574
Cites: 132
Cited by:
Related to: 10