FineSurE-VN: Semantic-Based Evaluation of Vietnamese Text Summarization
Year: 2026
Type: conference-paper
Abstract: Evaluating the quality of text summarization for Vietnamese is both critical and challenging, particularly as the demand for reliable automatic summarization grows in Vietnamese-language applications. This field has currently been constrained by traditional evaluation methods such as ROUGE and BLEU, which primarily focus on word matching and struggle to assess how effectively a summary conveys meaning and information. Vietnamese presents significant challenges because of its complex morphology, flexible word order, and nuanced semantic relationships. Consequently, traditional metrics often fail to accurately differentiate between summaries of varying quality, complicating robust evaluation. In this study, we apply the FineSurE framework semantics-based evaluation method that leverages large language models to evaluate Vietnamese news summarization. We use the ViMs dataset, comprising 1,945 Vietnamese news articles with expert-annotated summaries and key facts, to benchmark our approach. Experimental results indicate that, although the two summary variants concise and detailed differ significantly in quality, both achieve uniformly high ROUGE scores (all above 92%), which highlights the insensitivity of traditional metrics. In contrast, FineSurE provides much clearer scores, demonstrating that the detailed summaries are significantly more accurate (85.38% compared to 53.12%) and complete (42.39% compared to 13.75%). This underscores FineSurE's superior ability to identify important quality differences in summarizing Vietnamese texts.
Institutions: FPT University, Academy of Cryptography Techniques
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7203672385