A Monocular Fisheye Video-Based 2D to 3D Pose Lift Technique with Multiperson Spatial Context Integration
Year: 2024
Type: conference-paper
Abstract: In this paper, we propose a person intrinsic scaling method for our novel spatio-temporal encoding based 2D to 3D pose lifting technique from monocular fisheye video. While 2D pose estimation has advanced significantly, it lacks depth information and cannot capture out-of-plane movements. To handle this problem, 3D pose estimation is required. This paper proposes a framework for 2D to 3D pose lifting that tackles challenges in fisheye videos. Our method addresses fisheye deformation through a person's intrinsic scaling approach and leverages a transformer architecture with spatio-temporal self attention for pose lifting. We introduce a novel per-person scaling method to handle the depth ambiguity inherent in fisheye images. In a real life care facility data set, our framework outperformed Mediapipe, a state-of-the art method for pose lifting, in 4 different type of activity poses with a mean rmse, mae and mape value of 0.082, 0.047, and 0.046 respectively. This approach shows a new direction for 2D to 3D pose lifting technique.
Authors: Iqbal Hassan, Nazmun Nahid, Sozo Inoue
Institution: Kyushu Institute of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Universitas Brawijaya
Award: G799352828
OpenAlex ID: W4402721852