Acoustic-to-Articulatory Inversion Based on Externally Visible Articulator Trajectories and a Channel Attention Mechanism
Year: 2026
Type: article
Abstract: Acoustic-to-articulatory inversion (AAI) predicts articulatory movements from acoustic signals; however, current methods often overlook external, visible articulatory features and frequency domain information. Limited paired acoustic–articulatory data remain a practical constraint on robust AAI modeling and evaluation. This paper proposes a novel AAI framework integrating external visible articulatory features and a channel attention mechanism for feature fusion. We first construct a Mandarin EMA dataset for language learning, then explore the impact of visible features (e.g., lip movements) on predicting internal articulatory trajectories. We additionally investigate short-time Fourier magnitude representations computed from EMA-measured lip–jaw trajectories as auxiliary kinematic features. SENet- and ECA-based group-attention variants are evaluated as alternatives for reweighting input groups before concatenation. Experiments on SAIT-EMA and MOCHA show that visible articulatory features and frequency-domain information improve prediction accuracy, with channel attention fusion providing further gains in selected configurations. These findings demonstrate the value of attention-based multimodal data fusion for intelligent audio and articulatory information processing in pronunciation-learning contexts.
Source: Electronics
Authors: Yanlu Xie, Yujia Jin, Yifeng Sun, Xuhui Lan, Shiyao Zhu, Yingming Gao
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W7212139449