Year: 2024
Type: preprint
Abstract: The past few years have seen remarkable progress in the decoding of speech from brain activity, primarily driven by large single-subject datasets. However, due to individual variation, such as anatomy, and differences in task design and scanning hardware, leveraging data across subjects and datasets remains challenging. In turn, the field has not benefited from the growing number of open neural data repositories to exploit large-scale deep learning. To address this, we develop neuroscience-informed self-supervised objectives, together with an architecture, for learning from heterogeneous brain recordings. Scaling to nearly 400 hours of MEG data and 900 subjects, our approach shows generalisation across participants, datasets, tasks, and even to novel subjects. It achieves improvements of 15-27% over state-of-the-art models and matches surgical decoding performance with non-invasive data. These advances unlock the potential for scaling speech decoding models beyond the current frontier.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Exploring the Mechanisms of Distributed Spontaneous Brain Activity., G2792775314, Integrative imaging of brain structure and function in populations and individuals, Phonetic studies of Hawaiian speakers and learners, Brain Commands and Beyond: Decoding Inner Speech for Neural Prosthetics, EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems, G7299060899, Hermann Hesse and Pietism
OpenAlex ID: W4399455330