Comparing Automatic Metrics with Human Perceptions: the Ethical Challenges of Evaluating the Human-likeness of Chatbots
Year: 2025
Type: article
Abstract: Chatbots have become increasingly human-like, making it difficult for people to distinguish their outputs from those of people. Recently, an automated metric has been proposed to measure the level of anthropomorphism in the outputs of large language models (LLMs). However, it is not clear how well suited this is to assessing the human-likeness of conversational systems, such as chatbots. We build three LLM-based chatbots, and conduct a pilot user study to assess participants’ perceptions of their personification, comparing the results with the automated scores. We find that the human and automated measures are not well aligned, and that the levels of anthropomorhpism of the three systems vary.
Authors: Chen, Dongmin, Chen, Zihui, Hayes, Alexander, Jurcaga, Lukas, Karde, Prathamesh, Zhou, Yu, Dinkar, Tanvi, Abercrombie, Gavin
Language: English
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W6962768495
FWCI:
Cites:
Cited by:
Related to: