Work

Improving weakly supervised ASR from TV subtitles via refinement and adversarial training

Year: 2026

Type: article

Abstract: Automatic speech recognition (ASR) systems typically require large volumes of high-quality transcribed speech. In low-resource settings, where accurate verbatim transcripts are scarce, weakly supervised (WS) sources provide a practical alternative. In this work, we use the audio, subtitle texts, and subtitle timestamps from 16 television programs broadcast by the Vlaamse Radio- en Televisieomroeporganisatie (VRT, Flemish Radio and Television Broadcasting Organisation) as WS resources for ASR training. We examine the value of subtitles by comparing them with pseudo transcripts generated by existing ASR models, which are commonly used in WS ASR. Since TV subtitles are often imperfectly aligned with speech, we first propose a subtitle refinement pipeline that uses Whisper-based pseudo transcriptions, forced alignment, and subtitle-specific processing to produce better-aligned training pairs. We then introduce a more flexible end-to-end alternative based on adversarial training, which models text predictions at the distribution level and reduces the reliance on precise text–speech alignment. Experiments with different ASR backbones show that training on refined subtitles substantially improves recognition performance over raw subtitles and direct pseudo-transcript supervision. The adversarial approach achieves competitive overall performance while offering a simpler training framework for more weakly aligned settings.

Source: EURASIP Journal on Audio Speech and Music Processing

DOI: 10.1186/s13636-026-00478-x

Authors: Xinnian Zhao, Hugo Van Hamme

Institution: KU Leuven

Language: English

Topic: Speech Recognition and Synthesis

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $1,890

Funder: Fonds Wetenschappelijk Onderzoek