Work

Multimodal situation awareness for big data environments via adaptive sampling and dual self-distillation

Year: 2026
Type: article
Abstract: Situation awareness (SA) in disaster management requires robust modeling of multimodal big data from complex environments. With the increasing availability of large-scale video and audio data from surveillance systems, drones, and public sensors, multimodal approaches have become essential for improving reliability. However, existing methods often rely on uniform data sampling and treat modalities equally, limiting their effectiveness when data quality varies across time and modality. Compared with our previous work, this study extends from modality-specific analysis to a multimodal audio–visual framework. To address these challenges, this paper proposes MSA-SD², a multimodal situation awareness framework specifically designed for big data environments based on selective dual self-distillation. An adaptive temporal sampling and alignment module is introduced to prioritize informative temporal segments and ensure consistent audio–visual correspondence, enabling efficient processing of large-scale multimodal big data. For visual representation, a superpixels-guided vision transformer is employed, while a transformer-based audio encoder models temporal acoustic patterns. A multimodal student model is guided by visual and audio teacher models through a selective distillation strategy that adapts to modality reliability. Experimental results demonstrate improved performance on large-scale multimodal datasets, particularly under challenging big data conditions with varying data quality across modalities.
Language: English
Open Access status: gold
Estimated APC paid: $2,290
OpenAlex ID: W7234462424
FWCI:
Cites:
Cited by:
Related to: 10