Interpretable Modeling of Articulatory Temporal Dynamics from Real-Time MRI for Phoneme Recognition
Year: 2026
Type: conference-paper
Abstract: Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing. The source code is publicly available.1
Authors: Jay Park, Hong Nguyen, Sean Foley, Jihwan Lee, Yoonjeong Lee, Dani Byrd, Shrikanth Narayanan
Institution: University of Southern California
Language: English
Topic: Speech and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
OpenAlex ID: W4414700355
FWCI:
Cites:
Cited by:
Related to: