Work

The inadequacy of offline large language model evaluations: A need to account for personalization in model behavior

Year: 2025

Type: article

Abstract: Standard offline evaluations for language models fail to capture how these models actually behave in practice, where personalization fundamentally alters model behavior. In this work, we provide empirical evidence showcasing this phenomenon by comparing offline evaluations to field evaluations conducted by having 800 real users of ChatGPT and Gemini pose benchmark and other questions to their chat interfaces.

Source: Patterns

DOI: 10.1016/j.patter.2025.101397

Authors: Angelina Wang, Daniel E. Ho, Sanmi Koyejo

Institutions: Cornell Tech, Stanford University

Language: English

Topic: Topic Modeling

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $4,900

Funders: Schmidt Futures, National Science Foundation, John D. and Catherine T. MacArthur Foundation, Institute of Education Sciences