Year: 2022
Type: dissertation
Abstract: Long-form question answering (LFQA) is a challenging natural language processing task as it asks to produce paragraph-length answers to questions. Meaningful progress in LFQA would significantly speed up how fast people can seek information from large collections of data. This thesis aims to answer how feasible it is to bring LFQA to production, what are the practical ways to accomplish it, and what are the roadblocks for the application of its use. For this, we surveyed how neural QA methods can be applied to the task and developed a challenging abstractive long-form QA dataset to train LFQA models on. We then trained and evaluated BART and Longformer models on the dataset to test how well neural methods perform on the task. We conclude that progress in the task is hampered by the limited capabilities of neural networks to understand the text, shortcut-learning, limited datasets, and weaknesses in the evaluation metrics. The experimental part of the thesis demonstrates that the performance of the neural method on the abstractive QA dataset falls far behind the human baseline. The best performing neural models score low with automatic evaluation metrics, scoring 14.26 and 15.24 respectively on the ROUGE-L metric. In human evaluation, the neural model is considered accurate 23% of the time and preferred over human answers 19% of the time. Automatic evaluation metrics used in evaluation fail to show a correlation between how difficult the question and the automatic metric score (pROUGE−L = 0.54), supporting the thesis that they are insufficient for abstract LFQA evaluation. Furthermore, the models failed to use supporting external knowledge and mainly relied on parametric knowledge to answer questions, suggesting practical limitations of how used methods can integrate information from long and diverse text inputs.
Source: Aaltodoc (Aalto University)
Author: Antikainen, Simo
Institution: Aalto University
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7148225193