Improving Human Behavior Understanding with Semi-supervised Learning
Year: 2026
Type: dissertation
Abstract: Recent advancements in Computer Vision (CV) and Machine Learning (ML) have fundamentally revolutionized the field of human behaviour understanding, enabling intelligent systems to move beyond simple motion detection to the interpretation of complex, semantic actions from vast streams of visual data. This paradigm shift has been driven by the exponential growth of Deep Learning algorithms, particularly the evolution from 3D Convolutional Neural Networks (3D CNNs) to modern Vision Transformers (ViTs) , which have achieved remarkable success in modeling long-range spatiotemporal dependencies in RGB videos and multimodal signals. However, despite their impressive performance on controlled benchmarks, the deployment of these pixel-based models in real-world scenarios remains heavily constrained. Their reliance on high-dimensional visual data often incurs prohibitive computational costs that hinder real-time application on edge devices. Furthermore, these methods exhibit significant sensitivity to background clutter and viewpoint variations , and the processing of raw video raises critical privacy concerns by capturing unnecessary appearance details such as identity and clothing. Maximizing the potential of this modality presents a significant set of challenges that hinder widespread deployment. The heavy reliance on massive, high-quality annotated datasets creates a prohibitive bottleneck for supervised learning, while existing generative models used for data augmentation often lack physical plausibility or controllability. Furthermore, self-supervised methods borrowed from video domains frequently fail to capture the fine-grained dynamics of skeletal motion, and current evaluation metrics for synthetic data struggle to distinguish between realistic kinematics and artifacts. Addressing these limitations requires a fundamental shift toward more label-efficient, physically aware, and generative approaches to behaviour analysis. In this thesis, we proposed a comprehensive suite of novel solutions to mitigate these problems by exploiting semi-supervised and self-supervised learning paradigms, as well as advanced generative modeling. First, we addressed the limitations of temporal reasoning in Graph Convolutional Networks (GCNs) by proposing REGINA, a reasoning- augmented architecture. This method injects explicit temporal self-similarity cues into the network through a learnable temporal adjacency matrix, allowing the model to adaptively optimize its receptive fields to capture long-range dependencies that standard convolutions often miss. Addressing data scarcity, we proposed Kinetic-GAN, a generative adversarial network that scales action conditioning by an order of magnitude over prior works. By generating actions directly from a disentangled latent space using a mapping network and stochastic noise injection, it synthesizes diverse, high-fidelity global body movements across more than 100 action classes. We further advanced representation learning without labels through ATOM, a self-supervised strategy that leverages atomic motion representation learning. By decomposing actions into short, fine-grained primitives and training the backbone to recover their temporal order, ATOM learns robust features from the intrinsic structure of motion, transferring effectively to supervised tasks without the computational burden of heavy reconstruction heads. To bridge the gap between generation and recognition, we developed an intrinsic, kinematics-aware quality assessment framework. Recognizing that conventional distribution metrics fail to capture physical validity, this framework utilizes per-sample kinematic indicators within a weaklysupervised discriminator to iteratively filter synthetic samples, ensuring only physically plausible data is used to boost downstream performance. Finally, pushing the boundaries of controllable synthesis, we proposed GLDM, a Spatiotemporal Graph Latent Diffusion Model. GLDM projects skeleton sequences into a compact, graph-aligned latent space and performs diffusion with graph-aware updates. Crucially, by integrating cross-attention mechanisms, GLDM enables precise multi-modal conditioning, allowing generation to be guided not just by class labels but also by natural language prompts, achieving generalization beyond training data, unconstrained temporal flexibility and solving the mode collapse issues inherent to adversarial approaches. Results obtained by the proposed methods show that it is possible to improve HBU systems regarding label efficiency, generative controllability, and physical plausibility, while still obtaining state-of-the-art results. All proposed methods were evaluated on several large-scale benchmarks, demonstrating that robust behaviour understanding can be achieved with significantly reduced reliance on manual annotation.
Author: Degardin, Bruno Manuel
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225277943