Establishing best practices in large language model research: an application to repeat prompting
Year: 2024
Type: article
Abstract: OBJECTIVES: We aimed to demonstrate the importance of establishing best practices in large language model research, using repeat prompting as an illustrative example. MATERIALS AND METHODS: Using data from a prior study investigating potential model bias in peer review of medical abstracts, we compared methods that ignore correlation in model outputs from repeated prompting with a random effects method that accounts for this correlation. RESULTS: High correlation within groups was found when repeatedly prompting the model, with intraclass correlation coefficient of 0.69. Ignoring the inherent correlation in the data led to over 100-fold inflation of effective sample size. After appropriately accounting for this issue, the authors' results reverse from a small but highly significant finding to no evidence of model bias. DISCUSSION: The establishment of best practices for LLM research is urgently needed, as demonstrated in this case where accounting for repeat prompting in analyses was critical for accurate study conclusions.
Language: English
Subfield: Statistics, Probability and Uncertainty
Field: Decision Sciences
Domain: Social Sciences
Open Access status: hybrid
Estimated APC paid: $4,549
Funders: Stanford Institute for Human-Centered Artificial Intelligence, Stanford University, Gordon and Betty Moore Foundation, American Heart Association, National Institutes of Health, Stanford Bio-X, National Institute on Drug Abuse, National Institute of Allergy and Infectious Diseases, National Center for Advancing Translational Sciences
OpenAlex ID: W4405230597