Viet-LLM-Bench: Task-Specific Error Patterns and Judge Reliability in Vietnamese Evaluation of Chinese LLMs
Year: 2026
Type: preprint
Abstract: Vietnamese-language evaluation of large language models requires more than a single accuracy score: fluent responses can still misinterpret cultural meanings, mistranslate source text, or fail to follow instructions. We study 150 Vietnamese-language prompts in three task groups (Language_AI,Language_Data, and Translation_Data) answered by DeepSeek and Qwen in two runs, yielding 600 outputs. Run 1 was annotated by one human researcher and Run 2 by a Claude-based evaluator; 40 Run 2 outputs also received independent human annotations. We analyze correctness, naturalness, instruction compliance, error severity, and primary error type using prompt-aware permutation tests, multiplicity correction, and human–judge agreement measures. After Holm correction, no overall DeepSeek–Qwen difference was statistically robust. Task group was associated with several outcomes, especially compliance and severity; error-label distributions also differed across groups (Cramér’s V = 0.395 for Run 1 and 0.356 for Run 2; permutation p approximately 0.0005 in each). Human–Claude agreement on 40 responses was 92.5% for instruction compliance (kappa = 0.818) and 67.5% for primary error classification (kappa = 0.385). Because runs used different raters and prompts were purpose-built, theresults are exploratory rather than causal or a general model ranking. The benchmark illustrates why Vietnamese evaluation should report task-specific errors and validate automated judging by dimension.
Author: Lai, HaiAnh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225738144
Cites:
Cited by:
Related to: