Surgical Intelligence Labs: Collecting First-Person Surgical Data with a Smartphone to Train Autonomous Robots
Year: 2026
Type: preprint
Abstract: Background. Robot foundation models learn from large, diverse data, and human video is becoming a powerful way to supply it. Surgery has almost none beyond console video, yet the hands of surgeons, assistants and scrub technologists, seen from their own eyes, could be the main source to train autonomous robots in the operating room (OR). Methods. We built Surgical Intelligence Labs, a smartphone app that turns a clinician's iPhone into a first-person (egocentric) data-collection device. A head-mounted phone records tasks from operative field and back-table setup to console and microscope views, saving each frame with its 6-degree-of-freedom (6-DoF) head pose from Apple's ARKit and 50 Hz device motion. Hands are tracked on the phone (21 key points each), lifted to three dimensions by fitting an articulated hand model, and retargeted to a Unitree G1 + humanoid with five-finger Wuji Hand 2 hands and to ASIA, our bimanual scrub-technician robot. OR recording requires institutional approval, consent and on-device de-identification. Results. On a 63-s suturing exercise recorded with a head-mounted iPhone 17, ARKit tracked the head in all 1,877 video frames, with the phone a mean 55° below the horizon, inside the framing guide's target band. Both hands were tracked on the phone in 97% of sampled frames, and the 3D hand fit was kept for 99% of hand observations. Position-based retargeting to five-finger Wuji Hand 2 hands kept thumb-to-fingertip gaps in pinches and grips within 2 mm of the fitted human gaps (medians). Inverse kinematics solved all 300 G1 + and 200 WidowX test poses (median errors 1.1 and 0.35 mm). Conclusions. A consumer smartphone can capture first-person video, head pose, and hand motion that can be mapped to actions humanoid and bimanual robots can execute. These findings demonstrate feasibility; they do not yet establish dataset quality or robot performance. Data collection, training, and deployment are the next steps.
Source: Research Square
Authors: Mustafa Khan, João Bombardelli
Language: English
Subfield: Surgery
Field: Medicine
Domain: Health Sciences
Open Access status: green
OpenAlex ID: W7226035430
