Work

RegioZH: Unveiling Comprehension Disparities across Regional Chinese Varieties in LLMs

Year: 2026

Type: conference-paper

Abstract: Current evaluations often treat Chinese varieties as monolithic, obscuring failures for regional users.We introduce RegioZH, a 2,607item multiple-choice benchmark for regional lexical reasoning across Mainland China, Taiwan, Hong Kong, Singapore, and Malaysia.Built by region-specific annotators and expertvalidated, it targets localized idioms, slang, code-mixing, politeness conventions, and sociocultural expressions.Zero-shot results on eleven LLMs show large disparities, including a 29.4-point average accuracy gap between Taiwan and Singapore.Persona and structured chain-of-thought prompts for five models yield selective gains, suggesting that some errors stem from retrieval or framing rather than only missing knowledge.Representation, attention, calibration, and surface-feature analyses show a dissociation between marker attention and semantic grounding, overconfident forced-choice errors, and localized lexical density as a consistent difficulty predictor.Qualitative analysis finds terminology misunderstandings, sociocultural inference gaps, and cross-regional semantic divergence.RegioZH demonstrates that apparent Chinese proficiency can mask uneven understanding across regional varieties, motivating intra-language robustness evaluation alongside multilingual generalization.

Source: Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)

DOI: 10.18653/2026.findings-emnlp.793