Work

Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

Year: 2026

Type: conference-paper

Abstract: Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly.However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions.Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features.While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modalityspecific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy.To tackle these challenges, we propose R 2 ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery.Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge.To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data.Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources.Extensive experiments demonstrate that R 2 ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.1

Source: Proceedings of the conference - Association for Computational Linguistics. Meeting

DOI: 10.18653/v1/2026.acl-long.961

Authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, Zitong YU