Work

A Multimodal Framework for Integrating Motion Capture Monitoring into ASD Screening Utilizing XR Applications

Year: 2025

Type: article

Abstract: Full-body tracking (FBT) enhances embodiment and motion-based interaction in extended reality (XR), particularly for virtual reality (VR) applications such as those used for autism spectrum disorder (ASD) screening. While Meta Movement Software Development Kit (SDK) provides eye and hand tracking, its estimation-based FBT skeleton conflicts with Open Sound Control (OSC)-based body tracking data, leading to body proportion distortion and misalignment when external trackers like SlimeVR are used. To resolve this, we adopt the Virtual Motion Capture (VMC) protocol skeleton as the primary FBT framework, integrating OSC-based body tracking with OpenXR for head tracking and Meta’s SDK for eye and hand tracking. Multiple custom OSC messages are created by retrieving positional and rotational data from custom input actions, allowing real-time player headspace detection and precise avatar alignment. This approach ensures accurate skeletal proportions and real-time avatar alignment, eliminating reliance on Meta’s prediction-based body tracking. By leveraging custom OSC messages and SlimeVR’s body proportion calibration, the proposed system offers a more stable and precise FBT solution for standalone VR applications.

Source: Proceedings of the ACM on Human-Computer Interaction

DOI: 10.1145/3734435

Authors: Chin Yee Khor, Siti Azreena Mubin, Chin Ike Tan, Vinesh Thiruchelvam, Tse Kian Neo, Gaik Kin Teoh

Institutions: Asia Pacific University of Technology & Innovation, Multimedia University, IMU University

Language: English

Topic: Virtual Reality Applications and Impacts

Subfield: Human-Computer Interaction

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Ministry of Higher Education, Malaysia

Award: G382986582