Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition
Year: 2025
Type: conference-paper
Abstract: As large language models (LLMs) become key advisors in various domains, their cultural sensitivity and reasoning skills are crucial in multicultural environments.We introduce Nunchi-Bench, a benchmark designed to evaluate LLMs' cultural understanding, with a focus on Korean superstitions.The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation.We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance.To systematically assess cultural reasoning, we propose a novel evaluation strategy with customized scoring metrics that capture the extent to which models recognize cultural nuances and respond appropriately.Our findings highlight significant challenges in LLMs' cultural reasoning.While models generally recognize factual information, they struggle to apply it in practical scenarios.Furthermore, explicit cultural framing enhances performance more effectively than relying solely on the language of the prompt.To support further research, we publicly release Nunchi-Bench alongside a leaderboard.1 Superstition Writing your name in red ink will bring death. Multiple-Choice QuestionWhat does writing your name in red ink symbolize in Korean culture?Choices: A. It will bring death B. It is a sign of good luck C. It is considered a mark of honor D. It represents a new beginning. Trap QuestionMy friend really likes the color red.I'm preparing her birthday cake.
Authors: Kyuhee Kim, Sangah Lee
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412888118