Temporal Precision Matters: Brain-Tuning Speech Language Models with Millisecond-Resolution Neural Signals
Year: 2026
Type: conference-paper
Abstract: Brain-tuning enhances brain alignment and downstream performance by fine-tuning speech language models with neural recordings.However, previous work relies primarily on fMRI, whose temporal resolution integrates neural activity over seconds, blending distinct processing stages into a single supervision signal and precluding temporally targeted training.We introduce ECoG-tuning, which leverages electrocorticography's millisecond precision to train speech language models.We design temporally targeted windows-a speech window capturing acoustic-phonetic encoding and a language window capturing higher-order linguistic processing-grounded in neuroscientific findings about temporal encoding hierarchies.Evaluating three models on the Podcast ECoG dataset, we find that ECoG-tuning significantly improves brain alignment over pretrained and distillation baselines.Notably, full spatiotemporal dynamics yield 7-17% higher alignment than time-averaged supervision across models, and language-window tuning produces larger gains in higher-order language regions, indicating that temporal precision provides additional training value.Moreover, ECoG-tuned models consistently improve or maintain downstream performance.Overall, our work provides initial evidence that electrophysiology is a viable brain-tuning modality, demonstrating how neuroscientific insights into processing hierarchies can inform principled model training strategies.
Authors: Zhejun Zhang, Wenqing Zhou, Haozhe Xu, Lin Zhang, Lei Li
Institution: Beijing University of Posts and Telecommunications
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G780185718
OpenAlex ID: W7166819316