Evaluating Transformer-Based Architectures for Simultaneous Audio Speech Transcription and Background Audio Captioning
Year: 2025
Type: conference-paper
Abstract: This study evaluates transformer-based models for simultaneous speech transcription and background audio captioning in mixed audio scenarios. Using Whisper for speech and Prompteus for environmental sounds, the models are tested on the Clotho Voice dataset, which combines Portuguese speech (Common Voice 5.1) and environmental audio (Clotho 2.1). Results using WER and FENSE metrics show that each model performs well in its domain but degrades with overlapping signals. Whisper is robust to moderate noise, while Prompteus struggles with dominant speech. The findings highlight the need for hybrid approaches to enable reliable, context-aware audio processing in complex environments.
Authors: João Vitor R. da Silva, Francisco de Assis Boldt, Luis A. Souza Jr, Mariella Berger, Anselmo Frizera, Alberto F. De Souza, Thiago Oliveira-Santos, Claudine Badue
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G1555167115, G1773415617
OpenAlex ID: W4414227227