Freebie: A Multi-Camera System for Real-Time Markerless Hand Tracking in Non-Human Primates
Year: 2026
Type: preprint
Abstract: Objective. Neural decoders for dexterous finger movements are limited by the accuracy with which ground truth movement is measured. In non-human primates (NHPs), hand kinematics are typically recorded with instrumentation that constrains movement dimensionality. We evaluated whether a markerless camera system could replace conventional methods for tracking movements in real time. Approach. We developed Freebie, a real-time markerless system for tracking NHP finger movements. Keypoints detected online were used in place of manipulandum bend sensor signals. Two NHPs performed tasks by flexing fingers to move a virtual hand into target zones. The virtual hand was directly controlled using bend sensors or Freebie-derived kinematics, or indirectly controlled through a Kalman filter (KF) decoding kinematics from forearm electromyographic (EMG) activity. We quantified Freebie's accuracy relative to manual labels, agreement with bend sensor measurements, and latencies. We assessed online performance under Freebie control and compared EMG KFs trained using Freebie-derived labels with those trained using bend-sensor-derived labels. Main Results. We observed a median end-to-end latency of 15.8 ms at a 30 FPS camera acquisition rate, and 16.2 ms at 60 FPS. Freebie labels closely tracked bend sensor positions (median |r|=0.96). Median keypoint errors were 1.7 to 6.5 pixels, below the 20-pixel width of the smallest tracked finger. Direct Freebie control did not differ from bend sensor control on five of six online metrics; path efficiency was higher with Freebie. EMG KFs trained on Freebie-derived labels performed similarly to those trained on bend sensor labels online. When evaluated offline, models trained with Freebie had lower error. Significance. These findings indicate that bend sensors are not required to obtain finger labels for training a decoder: markerless keypoints alone provided effective labels. This is an important first step towards real-time tracking of more complex movements, beyond what could be easily measured with a manipulandum.
Institutions: University of Michigan, Scuola Superiore Sant'Anna
Language: English
Topic: Motor Control and Adaptation
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: green
Awards: EFRI BRAID: Neurally Inspired, Resilient Closed Loop Feedback Control of Learned Motor Dynamics, G3323308467, G3776160423, G5863455821, Graduate Research Fellowship Program (GRFP), NCS FR - Elucidating the relationship between motor cortex neural firing rates and dextrous finger movement EMG for use in brain computer interfaces
OpenAlex ID: W7225889762
