Type: dissertation
Abstract: This doctoral dissertation summarizes all my past work, including three novel Token Memory models that used the tokenizer dictionary to enhance model performance through token-level information augmentation.The paper is structured into four principal components:First, the study systematically reviews the evolutionary history of natural language processing (NLP) models, tracing developments from early symbolic and rule-based systems to contemporary Transformer architectures and large-scale pertaining models (LLM).This historical analysis establishes the theoretical foundation for our work, emphasizing two important technologies: Transformer, the predominant framework for modern language models, and Word2Vec, which inspired our core hypothesis that supplementary token-level information could enhance model capabilities.Second, we propose a Token Memory model for code generation tasks.This model employs the internet to search for token information and dynamically inject it into model computations.Evaluations demonstrate its superior performance compared to baseline models: 1.5× improvement over conventional Transformers of equivalent size, 3× enhancement against non-fine-tuned GPT-3, though remaining inferior to 25× larger SOTA models.Additionally, in strict accuracy metrics, our model achieved comparable results with both same-scale Transformers and 25× larger SOTA Transformers, while surpassing non-fine-tuned GPT-3 by an order of magnitude.Third, we implement a T5-based Token Memory variant where Token Memory is generated through encoder modules and subsequently integrated into decoder operations, providing additional information for generation.This model achieved SOTA performance on the PersonaChat dataset through both automated metrics and human evaluations.Finally, we develop a model based on an LLM that autonomously constructs and utilizes Token Memory during generation processes.Trained and evaluated on PG-19, C4-en, and BookSum datasets, our approach establishes new benchmark results for long-context language modeling, achieving optimal performance on 1 million token context windows.This breakthrough enables LLMs to handle theoretically unbounded context lengths, addressing limitations in current architectures while meeting growing demands for extended context processing capabilities.iii Time passes quickly like a white pony's shadow across a crevice.I have studied for seven years in Japan, and countless people have accompanied me during these seven years.First of all, I would like to thank my parents and grandparents for their financial support and warm family environment.This allows me to focus most of my energy on research.Thanks to all the professors.First, Professor Zhao gave me enough space and time to choose a direction I liked during my undergraduate period.Second, Professor Markov taught me how to build a deep learning server and NLP knowledge during my graduate period.Finally, Professor Fujita provided me with financial and technical support so that I could graduate smoothly during the Ph.D. period.Thanks to my friend Wang and coach Fukuzawa of JEXER Higashi-Koganei Gym.They made me start training so that I can stay healthy in my busy study life.Also, thanks to Shou, who helped me play the game and saved me tons of time.Thanks to the MEXT, the University of Aizu, and the Tokyo University of Agriculture and Technology.Not only did they provide a learning platform, but they
Author: Sun, Taize
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7154548641