Work

HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models

Year: 2023

Type: preprint

Abstract: Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce HI-TOM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2310.16755

Authors: He, Yinghui, Wu, Yufan, Jia, Yilin, Mihalcea, Rada, Chen, Yulong, Deng, Naihao

Institution: University of Michigan

Language: English

Topic: Child and Animal Learning Development

Subfield: Developmental and Educational Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: green

Funder: Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México

OpenAlex ID: W4387964152

Cites: 10

Cited by: 37

Related to: 10