Work

Completing the knowledge lifecycle for language models

Year: 2024
Type: dissertation
Abstract: In recent years, language models (LMs) have achieved remarkable success, driven by the highly effective Transformer architecture.Supported by in-context learning and language model scaling laws, we can transform nearly all natural language processing (NLP) tasks into language modeling problems, and then develop exceptionally large and powerful language models to solve them.However, current language models still face critical challenges, such as the issue of hallucination and the difficulty of training these models to adapt to frequent updates in knowledge.In this dissertation, we identify two main reasons as the root causes of these challenges: implicit knowledge representation and sparse knowledge distribution.First, the knowledge is represented implicitly as parameters in language models, which makes it challenging to identify, locate, and edit a specific piece of knowledge inside a language model.Additionally, real-world new knowledge is distributed in the huge amount of unstructured data sparsely, making it difficult to extract and saturate useful knowledge without an effective knowledge extraction system.In summary, significant flaws still exist despite the great success and achievements of the language model related research.In this thesis, we aim to tackle these challenges by completing a knowledge lifecycle, to establish a knowledge-oriented updating cycle for existing language models, enabling them to continuously improve by extracting new knowledge, eliciting existing internal knowledge, and subsequently integrating and updating the model itself.We propose three steps to enable the self-improvement ability of language models: Knowledge Extraction and Summarization, Knowledge Elicitation from Language Models, and Knowledge Update and Integration.First, we need to develop a powerful extraction system capable of extracting and summarizing useful knowledge from the vast amounts of real-world data.After this extraction, we create algorithms that can efficiently draw out the existing implicit knowledge embedded within language models.Finally, we merge this elicited knowledge with the newly extracted information, manage updates and resolve conflicts, and finally integrate the refined knowledge back to the model to complete an updating cycle.We have made a number of novel contributions given the goal of building such a kind of self-updating cycle.For extracting structured knowledge from textual documents, we build RESIN, the first cross-lingual multi-document multi-media information extraction system, that is able to process hundreds of multi-media document clusters at a scale and generate high-quality event graphs.For knowledge elicitation, we propose sparse latent typing (SLT), a pre-training objective designed to encourage language models to develop a struc-Completing this PhD thesis has been a transformative journey, and it would not be possible without the support and encouragement from a host of people whom I am thrilled to acknowledge here.First and foremost, I would like to express my deepest gratitude to my advisor, Prof. Heng Ji, for her unwavering support, insightful guidance, and persistent encouragement throughout my Ph.D. study.Her expertise and mentorship have profoundly influenced not only my academic growth but also enhanced my abilities in critical thinking, team leadership, time management, and effective communication.Under her guidance, I have learned to navigate multifaceted challenges, develop innovative approaches to research, and hone my skills in synthesizing complex information.Her unwavering dedication to my success has been a cornerstone of my academic journey, shaping me into a mature and capable NLP researcher.
Language: English
Open Access status: green
OpenAlex ID: W7220812376
Cites: 101
Cited by:
Related to: 10