Low-Latency Hand-Gesture Recognition for Real-Time Virtual MIDI Control Using MediaPipe and OpenCV on Consumer Apple Silicon
Year: 2026
Type: preprint
Abstract: Physical Musical Instrument Digital Interface (MIDI) controllers frequently present cost and portability barriers for independent musicians and digital audio developers. While camera-based gesture control offers a hardware-free alternative, computer vision pipelines often suffer from high inference latency and frame-rate instability, rendering them impractical for real-time musical performance where sub-50ms responsiveness is critical. This paper presents the architecture and performance evaluation of AirMIDI, a lightweight, real-time virtual MIDI controller and audio modulation engine built in Python using OpenCV and Google's MediaPipe Hands framework. Running exclusively on consumer-grade Apple M1 hardware without dedicated external GPUs or depth-sensing cameras, the system captures 21-point 3D hand landmarks from a standard 720p webcam feed and maps five distinct spatial gestures—Pinch, Slide, Raise, Glide, and Tilt—to continuous MIDI Control Change (CC), Pitchbend, and Note On/Off messages. Experimental evaluation demonstrates that the proposed pipeline achieves an average throughput of 44.6 frames per second (FPS) and a mean end-to-end processing latency of 22.4 ms under Single-Hand Lite configuration. Furthermore, the implementation of an Exponential Moving Average (EMA) spatial filter reduces unintended MIDI jitter by 94.8% during static hand holds. These findings confirm that sub-35ms vision-based multi-parameter musical expression is viable on accessible consumer hardware.
Source: Open Science Framework
Author: Shaurya Keshari
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7223472883
Cites:
Cited by:
Related to: