UMA-SPLIT: Unimodal Aggregation for Both English and Mandarin Non-Autoregressive Speech Recognition
Year: 2026
Type: conference-paper
Abstract: This paper proposes a unimodal aggregation (UMA) based nonautoregressive model for both English and Mandarin speech recognition. The original UMA explicitly segments and aggregates acoustic frames (with unimodal weights that first monotonically increase and then decrease) of the same text token to learn better representations than regular connectionist temporal classification (CTC). However, it only works well in Mandarin. It struggles with other languages, such as English, for which a single syllable may be tokenized into multiple fine-grained tokens, or a token spans fewer than 3 acoustic frames and fails to form unimodal weights. To address this problem, we propose allowing each UMA-aggregated frame map to multiple tokens, via a simple split module that generates two tokens from each aggregated frame before computing the CTC loss. Experiments verify the proposed model’s effectiveness: it outperforms other advanced non-autoregressive models, and even matches the hybrid CTC/attention autoregressive model on LibriSpeech (English, 2.22%/4.93% WER for test clean/other with the 149 M model) and on AISHELL-1 (Mandarin, 4.43% CER).
Authors: Ying Fang, Xiaofei Li
Institutions: Zhejiang University, Westlake University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417081705
FWCI:
Cites:
Cited by:
Related to: