An Interactive Desktop Application for Human Gesture-Controlled Music Generation Using Deep Learning-Based Hand Tracking
Year: 2026
Type: article
Abstract: This paper presents an interactive desktop application for human gesture-controlled music generation. The system uses a standard webcam and deep learning-based hand tracking to detect both hands and convert hand features into musical control actions. The application was developed in Python and uses OpenCV, MediaPipe, FluidSynth, MIDI synthesis, and WAV stem playback. It provides three modes. Free Play lets the user control notes and dynamics with hand movement. Song Generation lets the user guide melody, bass, and chords inside a beat-based structure. Stem Control lets the user activate or mute separate audio stems with finger gestures. Deep learning is used for hand tracking, while musical output is produced through rule-based mappings, MIDI synthesis, and prerecorded WAV stem playback. The system was tested with 30 voluntary participants. The average frame rate was 21.81 fps, and the average MediaPipe inference time was 20.80 ms. The average calibration score was 7.93 out of 8 checks. In Stem Control, an average of 4.80 out of five software-defined task criteria were met. The software-defined beat-drop timing condition was met for 16 of the 30 participants. Questionnaire scores were positive, with 4.57 for ease of use, 3.97 for naturalness, and 4.47 for reuse intention. The results show that webcam-based hand tracking can support interactive music control without requiring specialized hardware.
Source: Technologies
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,930
OpenAlex ID: W7225803022
FWCI:
Cites:
Cited by:
Related to: