LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations
Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents, yet evaluations focus primarily on task success rather than cultural appropriateness or evaluator reliability.We introduce LIVECULTUREBENCH 1 , a multi-cultural, dynamic benchmark that embeds LLMs as agents in a simulated town and evaluates them on both task completion and adherence to socio-cultural norms.The simulation models a small city as a location graph with synthetic residents having diverse demographic and cultural profiles.Each episode assigns one resident a daily goal while others provide social context.An LLM-based verifier generates structured judgments on norm violations and task progress, which we aggregate into metrics capturing task-norm trade-offs and verifier uncertainty.Using LIVECULTUREBENCH across models and cultural profiles, we study (i) cross-cultural robustness of LLM agents, (ii) how they balance effectiveness against norm sensitivity, and (iii) when LLM-as-a-judge evaluation is reliable for automated benchmarking versus when human oversight is needed.
Institution: Monash University
Language: English
Subfield: Safety Research
Field: Social Sciences
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7166821057