Work

Evaluating Large Language Models through Communication Games: An Agent-Based Framework Using Werewolf in Unity

Year: 2025

Type: conference-paper

Abstract: In this study, we explore the reasoning capabilities of Large Language Models (LLMs) within the context of the social communication game Werewolf, aiming to evaluate their performance in managing complex system states commonly found in computer games.Our agent architecture gathers data, refines them into detailed information, and plans actions based on this knowledge.To demonstrate the feasibility of using LLM based agents in computer games, we developed a simulation and evaluation tool using the Unity game engine.This software enables users to experiment with various LLMs and agent architectures and to measure model performance within the application.For evaluation, we tested three models: GPT-3.5 Turbo, Mistral-7B-OpenOrca, and Nous-Hermes-Llama2-13B.The results show that even smaller models can perform reasonably well in Werewolf.However, their error rate is significantly higher, highlighting the need for additional software modules or fine-tuning to improve their accuracy.

Source: International Conference on the Foundations of Digital Games (FDG)

DOI: 10.1145/3723498.3723702

Authors: Christian Poglitsch, Fabian Szakács, Johanna Pirker

Institution: Graz University of Technology

Language: English

Topic: Artificial Intelligence in Games

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Austrian Science Fund

Awards: G2284405116, G4360184155