Work

Manual and automatic evaluation of summaries

Year: 2002
Type: conference-paper
Abstract: In this paper we discuss manual and automatic evaluations of summaries using data from the Document Understanding Conference 2001 (DUC-2001). We first show the instability of the manual evaluation. Specifically, the low inter-human agreement indicates that more reference summaries are needed. To investigate the feasibility of automated summary evaluation based on the recent BLEU method from machine translation, we use accumulative n-gram overlap scores between system and human summaries. The initial results provide encouraging correlations with human judgments, based on the Spearman rank-order correlation coefficient. However, relative ranking of systems needs to take into account the instability.
Language: English
Open Access status: gold
OpenAlex ID: W2160204597
FWCI: 6.715
Cites: 1
Cited by: 269
Related to: 10