Work

Geometry-Aware 3D Hand–Object Pose Estimation Under Occlusion via Hierarchical Feature Decoupling

Year: 2025

Type: article

Abstract: Hand–object occlusion poses a significant challenge in 3D pose estimation. During hand–object interactions, parts of the hand or object are frequently occluded by the other, making it difficult to extract discriminative features for accurate pose estimation. Traditional methods typically extract features for both the hand and object from a single image using a shared backbone network. However, this approach often results in feature contamination, where hand and object features are mixed, especially in occluded regions. To address these issues, we propose a novel 3D hand–object pose estimation framework that explicitly tackles the problem of occlusion through two key innovations. While existing methods rely on a single backbone for feature extraction, our framework introduces a feature decoupling strategy that shares low-level features (using ResNet-50) to capture interaction contexts, while separating high-level features into two independent branches. This design ensures that hand-specific features and object-specific features are processed separately, reducing feature contamination and improving pose estimation accuracy under occlusion. Recognizing the correlation between the hand’s occluded regions and the object’s geometry, we introduce the Hand–Object Cross-Attention Transformer (HOCAT) module. Unlike traditional attention mechanisms that focus solely on feature correlations, the HOCAT leverages the geometric stability of the object as prior knowledge to guide the reconstruction of occluded hand regions. Specifically, the object features (key/value) provide contextual information to enhance the hand features (query), enabling the model to infer the positions of occluded hand joints based on the object’s known structure. This approach significantly improves the model’s ability to handle complex occlusion scenarios. The experimental results demonstrate that our method achieves significant improvements in hand–object pose estimation tasks on publicly available datasets such as HO3D V2 and Dex-YCB. On the HO3D V2 dataset, the PAMPJPE reaches 9.1 mm, the PAMPVPE is 9.0 mm, and the F-score reaches 95.8%.

Source: Electronics

DOI: 10.3390/electronics14051029

Authors: Yuting Cai, Huimin Pan, Jiayi Yang, Yichen Liu, Quanli Gao, Xihan Wang

Institution: Xi'an Polytechnic University

Language: English

Topic: Human Pose and Action Recognition

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $2,896

Funder: National Natural Science Foundation of China