Year: 2020
Type: conference-paper
Abstract: Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models.GPT-2 (Radford et al., 2019), in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored.Our work takes a step in understanding GPT-2's outputs in terms of discourse coherence.We perform a comprehensive study on the validity of explicit discourse relations in GPT-2's outputs under both organic generation and fine-tuned scenarios.Results show GPT-2 does not always generate text containing valid discourse relations; nevertheless, its text is more aligned with human expectation in the fine-tuned scenario.We propose a decoupled strategy to mitigate these problems and highlight the importance of explicitly modeling discourse information.
Authors: Wei-Jen Ko, Junyi Jessy Li
Institution: The University of Texas at Austin
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: National Science Foundation
OpenAlex ID: W3095019873