Year: 2025
Type: dissertation
Abstract: Artificial Intelligence (AI) systems are increasingly integral to everyday life, demanding NLP models that can handle large-scale knowledge with both speed and reliability. Modern pretrained language models and semi-parametric architectures have significantly advanced Knowledge-Intensive Language Tasks (KILT), yet key challenges remain in managing external knowledge efficiently and curbing biases introduced by training data. First, we present an adaptive computation approach for Open-Domain Question Answering (ODQA). It dynamically focuses neural reading on the most promising retrieved documents, eliminating redundant processing of irrelevant text. This framework substantially reduces overall computational overhead while maintaining strong accuracy on benchmarks like Open-SQuAD. Second, we streamline KILT tasks by storing knowledge in synthetic QA pairs rather than full articles. We introduce PAQ (Probably Asked Questions), a comprehensive repository of automatically generated QA pairs from Wikipedia, offering broad coverage across diverse topics. Instead of retrieving lengthy documents, systems can directly query these pre-computed pairs for answers, achieving an efficient balance between speed and accuracy. We then propose Efficient Memory-Augmented Transformers (EMAT), which integrate key-value lookups into a single forward pass to deliver fast, knowledge-intensive generation. Third, we address biases that lead models to rely on spurious shortcuts. Our synthetic data generation strategies create debiased training corpora for tasks like natural language inference and fact-checking (e.g., SNLI, MNLI, FEVER). By rebalancing problematic examples and removing shortcuts, we foster models that are more robust to adversarial and out-of-domain settings. When combined with specialized debiasing techniques, these strategies yield state-of-the-art results on challenging test sets such as SNLI-hard and MNLI-hard. Collectively, these contributions illustrate that KILT systems can be both efficient and robust. We evaluate these methods extensively, demonstrating significant improvements across multiple knowledge-intensive benchmarks. By harmonizing computational cost, knowledge representation, and data quality, we move closer to realizing NLP models that seamlessly handle large-scale information while remaining robust to biases and superficial cues—an essential step toward truly knowledgeable AI.
Author: Wu, Yuxiang
Institution: University College London
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7071910417