Audio-visual speech enhancement with multi-level feature deep fusion under low signal-to-noise ratio
Year: 2025
Type: article
Abstract: To address the limitations in feature extraction and cross-modal fusion in audio-visual speech enhancement, a multistage deep fusion method was proposed for low signal-to-noise ratio (SNR) conditions. The method consisted of an audio-visual encoding network, a fusion network, and an auditory decoding network. A multi-branch collaborative unit (MCU) was introduced in the auditory encoder, along with an audio-visual attention fusion module (AVAFM) between each visual and auditory layer. A fusion weighting block (FWB) was also designed to optimize and dynamically weight features at each stage. Experiments on TMSV and LGRID datasets showed that the proposed method significantly improved PESQ and STOI scores under various low-SNR conditions. Compared to audio-only enhancement, average gains of 38.95% in PESQ and 33.92% in STOI were achieved at -5 dB, -2 dB, and 1 dB. These results demonstrate the method’s strong denoising ability and the effectiveness of visual information.
Authors: ZHANG Tianqi, SHEN Xiwen, TANG Juan, TAN Shuang
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7008080876
FWCI:
Cites:
Cited by:
Related to: