Work

High-Fidelity Neural Speech Reconstruction through an Efficient Acoustic-Linguistic Dual-Pathway Framework

Year: 2026

Type: preprint

Abstract: Reconstructing speech from neural recordings is crucial for understanding speech coding and developing brain-computer interfaces (BCIs). However, existing methods trade off acoustic richness (pitch, prosody) for linguistic intelligibility (words, phonemes). To overcome this limitation, we propose a dual-path framework to concurrently decode acoustic and linguistic representations. The acoustic pathway uses a long-short term memory (LSTM) decoder and a high-fidelity generative adversarial network (HiFi-GAN) to reconstruct spectrotemporal features. The linguistic pathway employs a transformer adaptor and text-to-speech (TTS) generator for word tokens. These two pathways merge via voice cloning to combine both acoustic and linguistic validity. Using only 20 minutes of electrocorticography (ECoG) data per subject, our approach achieves highly intelligible synthesized speech (mean opinion score = 4.0/5.0, word error rate = 18.9%). Our dual-path framework reconstructs natural and intelligible speech from ECoG, resolving the acoustic-linguistic trade-off.

Source: eLife

DOI: 10.7554/elife.109400.2

Authors: Jiawei Li, Chunxu Guo, Chao Zhang, Edward F Chang, Yuanning Li

Institutions: ShanghaiTech University, Shanghai Artificial Intelligence Laboratory, Tsinghua University, University of California System

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,500