Unified Spatio-temporal Vision Transformer Framework for Real-time Multi-modal Video Understanding and Event Prediction Leveraging Cross-attention Fusion and Self-supervised Learning for Complex Scene Interpretation in Dynamic Environments
Year: 2025
Type: article
Abstract: The rapid advancement of deep learning architectures has revolutionized video understanding, especially in dynamic, real-time applications such as autonomous vehicles, surveillance systems, and sports analytics. However, current models often struggle to jointly model spatial-temporal dynamics and cross-modal dependencies, leading to suboptimal performance in complex scenes. This paper presents a unified spatio-temporal vision transformer framework that integrates self-supervised learning with cross-attention fusion for multi-modal video understanding and event prediction. The proposed system employs a transformer-based encoder to capture spatio-temporal relationships, enhanced through fusion mechanisms that process RGB frames, motion cues, and audio signals. Self-supervised learning techniques ensure better generalization in real-world scenarios with limited annotations. Extensive benchmarking against state-of-the-art models demonstrates superior accuracy, robustness, and interpretability. This research provides a scalable and efficient solution for complex scene understanding, capable of real-time deployment in edge environments. Performance metrics show a substantial increase in predictive accuracy, supported by reduced training loss and improved F1 scores across test datasets.
Author: Researcher
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6911996280
FWCI:
Cites:
Cited by:
Related to: