Year: 2025
Type: conference-paper
Abstract: Current methods for human-object interaction segmentation excel in closed-world settings but struggle to generalize to open-world scenarios where novel actions emerge. Since collecting exhaustive training data for all possible dynamic human activities is impractical, a model capable of detecting and segmenting novel, out-of-distribution (OOD) actions without manual annotation is needed. To address this, we formally define the open-world action segmentation problem and propose a novel framework featuring three key components: 1) an Enhanced Pyramid Graph Convolutional Network with a new decoder for robust spatiotemporal upsampling, 2) hybrid-based training synthesizing OOD data to eliminate reliance on manual labels, and 3) a temporal clustering loss that groups in-distribution actions while distancing OOD samplesWe evaluate our framework on two challenging human-object interaction recognition datasets: Bimanual Actions and Two Hands and Object datasets. Experimental results demonstrate significant improvements over state-of-the-art action segmentation models across multiple open-set evaluation metrics, achieving 16.9% and 34.6% relative gains in open-set segmentation (F1@50) and out-of-distribution detection performances (AUROC), respectively. Additionally, we conduct an in-depth ablation study to assess the impact of each proposed component, identifying the optimal framework configuration for open-world action segmentation.
Authors: Hao Xing, Kai Zhe Boey, Gordon Cheng
Institution: Technical University of Munich
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4416750037