Year: 2021
Type: conference-paper
Abstract: Reference-based automatic evaluation metrics are notoriously limited for NLG due to their inability to fully capture the range of possible outputs.We examine a referenceless alternative: evaluating the adequacy of English sentences generated from Abstract Meaning Representation (AMR) graphs by parsing into AMR and comparing the parse directly to the input.We find that the errors introduced by automatic AMR parsing substantially limit the effectiveness of this approach, but a manual editing study indicates that as parsing improves, parsing-based evaluation has the potential to outperform most reference-based metrics.
Authors: Emma Manning, Nathan Schneider
Institution: Georgetown University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W3213775472