HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation
Year: 2025
Type: conference-paper
Abstract: Neural transducers (NT) provide an effective framework for speech streaming, demonstrating strong performance in automatic speech recognition (ASR).However, the application of NT to speech translation (ST) remains challenging, as existing approaches struggle with word reordering and performance degradation when jointly modeling ASR and ST, resulting in a gap with attention-based encoder-decoder (AED) models.Existing NT-based ST approaches also suffer from high computational training costs.To address these issues, we propose HENT-SRT (Hierarchical Efficient Neural Transducer for Speech Recognition and Translation), a novel framework that factorizes ASR and translation tasks to better handle reordering.To ensure robust ST while preserving ASR performance, we use self-distillation with CTC consistency regularization.Moreover, we improve computational efficiency by incorporating best practices from ASR transducers, including a downsampled hierarchical encoder, a stateless predictor, and a pruned transducer loss to reduce training complexity.Finally, we introduce a blank penalty during decoding, reducing deletions and improving translation quality.Our approach is evaluated on three conversational datasets Arabic, Spanish, and Mandarin achieving new state-of-the-art performance among NT models and substantially narrowing the gap with AED-based systems.
Institution: Johns Hopkins University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4412944384