Work

STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos

Year: 2023

Type: conference-paper

Abstract: We address the problem of extracting key steps from un-labeled procedural videos, motivated by the potential of Augmented Reality (AR) headsets to revolutionize job training and performance. We decompose the problem into two steps: representation learning and key steps extraction. We propose a training objective, Bootstrapped Multi-Cue Contrastive (BMC2) loss to learn discriminative representations for various steps without any labels. Different from prior works, we develop techniques to train a light-weight temporal module which uses off-the-shelf features for self supervision. Our approach can seamlessly leverage information from multiple cues like optical flow, depth or gaze to learn discriminative features for key-steps, making it amenable for AR applications. We finally extract key steps via a tunable algorithm that clusters the representations and samples. We show significant improvements over prior works for the task of key step localization and phase classification. Qualitative results demonstrate that the extracted key steps are meaningful and succinctly represent various steps of the procedural tasks. Our code can be found at https://github.com/anshulbshah/STEPs.

Source: IEEE/CVF International Conference on Computer Vision (ICCV)

DOI: 10.1109/iccv51070.2023.00952

Authors: Anshul Shah, Benjamin Lundell, Harpreet Sawhney, Rama Chellappa

Institutions: Johns Hopkins University, Microsoft (United States)

Language: English

Topic: Video Analysis and Summarization

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Multidisciplinary University Research Initiative, Office of Naval Research

Awards: G3587679958, G5121974832, G8006577219, G8876996369