Work

Application of Signal Detection Theory in Evaluating Trust of Information Produced by Large Language Models

Year: 2025

Type: article

Abstract: Hallucinations are a major threat in applying generative artificial intelligence technologies like large language models (LLMs) to high-consequence domains like nuclear electricity generation. Inappropriately trusting an LLM can have deleterious consequences, such as misusing unreliable LLMs and disusing reliable LLMs. Identifying methods to evaluate the trustworthiness of LLMs and resultant user trust is an open area that demands future research from both a technical perspective and a human factors perspective. In this paper, we highlight the challenges in evaluating trust in LLMs and then introduces SDT as a potential framework that may overcome these challenges. We then propose using signal detection theory (SDT) as an evaluative framework for comparing trust and trustworthiness when interacting with LLMs in high-consequence domains like nuclear electricity generation.

Source: Proceedings of the Human Factors and Ergonomics Society Annual Meeting

DOI: 10.1177/10711813251368829

Authors: Casey Kovesdi, Yusuke Yamani, Austin Jackson, Linyu Lin

Institutions: Idaho National Laboratory, Old Dominion University

Language: English

Topic: Human-Automation Interaction and Safety

Subfield: Social Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: bronze

Funder: National Energy Technology Laboratory

Award: G3430522758