Summarization from Leaderboards to Practice: Choosing A Representation Backbone and Ensuring Robustness
Year: 2023
Type: preprint
Abstract: Academic literature does not give much guidance on how to build the best possible customer-facing summarization system from existing research components. Here we present analyses to inform the selection of a system backbone from popular models; we find that in both automatic and human evaluation, BART performs better than PEGASUS and T5. We also find that when applied cross-domain, summarizers exhibit considerably worse performance. At the same time, a system fine-tuned on heterogeneous domains performs well on all domains and will be most suitable for a broad-domain summarizer. Our work highlights the need for heterogeneous domain summarization benchmarks. We find considerable variation in system output that can be captured only with human evaluation and are thus unlikely to be reflected in standard leaderboards with only automatic evaluation.
Source: arXiv (Cornell University)
Authors: Demeter, David, Agarwal, Oshin, Igeri, Simon Ben, Sterbentz, Marko, Molino, Neil, Conroy, John M., Nenkova, Ani
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4381558659