Year: 2025
Type: preprint
Abstract: Theory of Mind (ToM) is essential for interpreting beliefs, intentions, and emotions that may differ from reality. Previous studies have examined the capabilities of large language models (LLMs) in ToM tasks, highlighting both strengths and limitations. In this study, we compared reasoning-oriented and non-reasoning models on a ToM benchmark. For reasoning evaluation, we adopted a stepwise assessment approach. Since there are no ground-truth reasoning annotations for ToM, we employed two established evaluation frameworks: ROSCOE and ReCEval, to assess the relevance, coherence, and informativeness of reasoning steps. Our results show that GPT-4 produced the highest accuracy for the non-reasoning ToM task. In contrast, the ROSCOE evaluation indicated that the reasoning steps generated by DeepSeek-R1-671B were semantically and logically consistent, while the ReCEval results showed that OpenAI-o1 produced reasoning that was both correct and highly informative. The ReCEval and ROSCOE metrics revealed important reasoning behaviors, with ReCEval consistently producing higher scores while ROSCOE provided more granular analysis, revealing that some of the models can correctly predict an outcome, while struggling with reasoning articulation.
Institution: University of Aizu
Language: English
Subfield: Developmental and Educational Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W4413638528