Year: 2025
Type: dissertation
Abstract: The de facto paradigm of developing NLP models requires collecting a dataset, training the model on the training set assuming all the examples are i.i.d. (independent and identically distributed), and evaluating it on another evaluation set with similar data distribution. However, as recent larger models can (almost) fit the entire dataset, we are starting to observe many problems with the aforementioned paradigm. This thesis studies the reliability and generalization problems hidden beneath the aforementioned paradigm. We start by pointing out important evaluation problems and notice substantial performance instability problems from common NLP robustness benchmark scores. We conduct a thorough empirical analysis to quantify and locate this instability issue, showing how high inter-example correlations within the evaluation sets can lead to high instability. In addition to better evaluation, this thesis also focuses on developing new methods to improve generalization and provide a better understanding of recent progress on generalizability from large pretrained models. For compositional generalization, we develop data augmentation and unlikelihood-based learning methods to improve the model’s generalizability by encouraging the mutual exclusivity bias and providing more primitive-level supervision in the training set. To explain the recent success of large pretrained models, we present two pieces of work explaining the important ingredients of their success. Masked Language Modeling (MLM) is a popular pretraining objective for Transformer-based models. We show this objective can be used for POS-tag induction, linking its success to a strong foundation of syntactic understanding. Lastly, another factor behind the success of large pretrained models is the huge pertaining dataset. To further explain the importance of data, we provide an empirical study on the influence of several data factors for compositional generalization, showing how the scale, complexity, and difficulty of data can substantially impact the generalization performance.
DOI: 10.17615/hvyn-7z77
Author: Zhou, Xiang
Institution: University of North Carolina at Chapel Hill
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7077489104
Cites:
Cited by:
Related to: