Work

A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs

Year: 2016

Type: conference-paper

Abstract: We introduce a manually-created, multireference dataset for abstractive sentence and short paragraph compression.First, we examine the impact of single-and multi-sentence level editing operations on human compression quality as found in this corpus.We observe that substitution and rephrasing operations are more meaning preserving than other operations, and that compressing in context improves quality.Second, we systematically explore the correlations between automatic evaluation metrics and human judgments of meaning preservation and grammaticality in the compression task, and analyze the impact of the linguistic units used and precision versus recall measures on the quality of the metrics.Multi-reference evaluation metrics are shown to offer significant advantage over single reference-based metrics.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/d16-1033

Authors: Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi

Institutions: Microsoft (United States), University of Amsterdam

Language: English

Topic: Advanced Text Analysis Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Microsoft Research

OpenAlex ID: W2511538013