Year: 2023
Type: dissertation
Abstract: Recent research on transformer-based large language models and large-scale pretraining has shown unprecedented success in the field of natural language processing and beyond.A slew of previous work demonstrated that the rich knowledge embedded in large language models can be effective transferred across different modalities, to solve a wider range of discrinimative and generative tasks, e.g., vision-language understanding and robotic planning.In this thesis, we investigate an uncharted area of cross-modal transfer with large language models, decoding brain signals.State-of-the-art brain-to-text systems have achieved initial success in decoding language directly from brain signals using modern machine learning models such as recurrent neural networks.However, current approaches are limited to small closed vocabularies which are far from enough for natural communication.Additionally, most of the high-performing approaches require data from invasive devices (e.g., ECoG).To address the existing limitations, we extend the brain-to-text decoding problem to a more challenging open vocabulary setting, where the decoding is performed on the entire English vocabulary that is 100x larger than previous work.Furthermore, we decode from Electroencephalography (EEG) data which is non-invasive but more noisy.To handle the huge vocabulary size with a small amount of parallel EEG-text data, we propose a novel model leveraging pretrained large language models.The key idea is to learn a good brainlanguage alignment between the "brain embedding space" and the "pretrained language embedding space" so that we can directly leverage the power of a large language model, e.g., BART.Results show promising EEG-to-Text decoding performance with a BLEU-1 score of 40.1%.Moreover, in-depth analysis shows that the proposed model has strong scalability and robustness with respect to the data source, which is essential towards building a large-scale brain-language foundation model.To further demonstrate the effectiveness of the EEG-to-Text decoder, we propose a novel zero-shot plug-and-play framework and evaluate it on a challenging downstream task, EEGbased sentiment classification, where the task is to predict the sentence's sentiment based solely on the EEG signals.We show that our zero-shot framework can outperform fullysupervised baselines with a large margin.The code is made publicly available for research purpose at https://github.com/MikeWangWZHL/EEG-To-Text.First and foremost, I would like to express my utmost gratitude to my advisor, Professor Heng Ji, who has always been an excellent mentor and a good friend to me.She has provided me with endless
Author: Wang, Zhenhailong
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
OpenAlex ID: W7220806852