Year: 2025
Type: conference-paper
Abstract: Information Retrieval systems are becoming increasingly multimodal, and one interesting and natural modality for human information interaction is audio. Audio data is rapidly expanding in both volume and popularity. For example, more than 4 million podcasts are available worldwide - each containing tens to thousands of episodes - with over 500 million expected listeners as of 2024. Searching such large and evolving audio corpora involves multiple challenges, ranging from content and style diversity, expensive audio processing, and various indexing considerations such as retrieval segmentation and text vs audio retrieval modalities. In this thesis, we aim to address the challenges linked to audio retrieval and devise novel approaches to improve state-of-the-art (SOTA) performance. We divide the work into four main milestones that target answering the following research questions: RQ1: How does the quality of an ASR model affect retrieval effectiveness? One of the approaches to search spoken audio is to reduce the problem to text retrieval using ASR models to generate text transcripts. Spina et al. [2] showed that the quality of ASR has a clear impact on the summaries generated by ASR. Motivated by this, we are interested in examining how automatic transcription algorithms impact the effectiveness of audio retrieval and re-ranking in dense and sparse retrieval configurations. In many real-world scenarios, resource limits require cheaper ASR models to be used. Therefore, we are interested in exploring if dense semantic search methods can reduce the impact of ASR model choice. RQ2: How can we build more complete, comprehensive, and representative audio retrieval resources to promote further research on audio retrieval? After reviewing the literature and performing a preliminary experiment on the Spotify podcasts dataset [1], we noted two main gaps in the available resources. Firstly, the large audio dataset released in TREC suffers from the shallow pooling issue, where new systems retrieve many unseen judgments, making it difficult to assess the quality of modern IR techniques. Secondly, there is a scarcity of audio datasets available for low-resource languages. Our plan to bridge these gaps comprises two parts: (1) Assessing the missed judgments Motivated by the recent literature on using LLMs as a judge [3], we plan to resolve the unjudged passages issue by assessing the relevance with multiple large language models (LLMs). (2) Curate a multi-lingual audio dataset with a focus on low-resource languages. Since there is a scarcity of audio datasets in low-resource languages, we plan to collect a large number of audio files in multiple low-resource languages, unify their format, and make them publicly available as a first step toward building test collections that enable research and advancements in such languages. RQ3: How can we combine audio and text to produce an efficient and effective multi-modal ranking model? We will focus on solutions that depend on processing both text and audio signals during indexing and retrieval. Our first study will be focused on improving the effectiveness of low-cost ASR systems. Another promising direction is to use the phonetic features as a representation of a word or token. Then, we can devise a suitable indexing paradigm to support searching over these representations. RQ4: What are the challenges affecting audio indexing and search performance, and what is the best way to mitigate against similar issues in the future? One of the main challenges in multi-modal search is identifying the suitable length for the retrieval unit. The retrieval unit satisfying an information need might vary in length from a couple of minutes to 10 minutes to one hour or a whole episode or series. Other challenges will stem from failure cases within state-of-the-art text/audio retrieval systems. So, in parallel with answering the previous questions, we will also study such failure cases, investigate the reasons behind them, and propose solutions that can mitigate their negative impacts.
Author: Watheq Mansour
Institution: The University of Queensland
Language: English
Topic: Music and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4412378174