Work

VIVAR: learning view-invariant embedding for video action recognition

Year: 2025

Type: article

Abstract: Deep learning has achieved state-of-the-art video action recognition (VAR) performance by comprehending action-related features from raw video. However, these models often learn to jointly encode auxiliary view (viewpoints and sensor properties) information with primary action features, leading to performance degradation under novel views and security concerns by revealing sensor types and locations. Here, we systematically study these shortcomings of VAR models and develop a novel approach, VIVAR, to learn view-invariant spatiotemporal action features removing view information. In particular, we leverage contrastive learning to separate actions and jointly optimize adversarial loss that aligns view distributions to remove auxiliary view information in the deep embedding space using the unlabeled synchronous multiview (MV) video to learn view-invariant VAR system. We evaluate VIVAR using our in-house large-scale time synchronous MV video dataset containing 10 actions with three angular viewpoints and sensors in diverse environments. VIVAR successfully captures view-invariant action features, improves inter and intra-action clusters’ quality, and outperforms SoTA models consistently with 8% more accuracy. We additionally perform extensive studies with our datasets, model architectures, multiple contrastive learning, and view distribution alignments to provide VIVAR insights. We open-source our code and dataset to facilitate further research in view-invariant systems.

Source: Maryland Shared Open Access Repository (USMAI Consortium)

DOI: 10.13016/m2oyws-ooca

Authors: Hasan, Zahid, Ahmed, Masud, Faridee, Abu Zaher Md, Purushotham, Sanjay, Lee, Hyungtae, Kwon, Heesung, Roy, Nirmalya

Language: English

Topic: Human Pose and Action Recognition

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Office of Naval Research, U.S. Army Combat Capabilities Development Command, Army Research Laboratory, DEVCOM Army Research Laboratory

Awards: G2071422347, EAGER: CNS: RobSenCom: A Middleware to Improve the Connectivity between Heterogeneous Robots and IoT, G5095998002, REU Site: Research Experiences for Undergraduates in Smart Computing and Communications, G6671297155, G7149300326, CAREER: Scalable and Adaptable Cross-Domain Autonomous Health Assessment, G8876996369

OpenAlex ID: W6884838551

FWCI:

Cites:

Cited by:

Related to: