Gesture talk: an Integrated Multimodal AI Assistant (Gesture, Voice, and Conversational Intelligence)
Year: 2025
Type: article
Abstract: Emerging trends in Human‑Computer Interaction (HCI) emphasize multimodal input systems that combine visual gestures, voice commands, and dialogue-based AI. This work presents a Python‑based assistant integrating MediaPipe/OpenCV, voice automation through SpeechRecognition and system subprocesses, and a Generative AI chatbot powered by Google’s Gemini API. Inspired by prior multimodal studies and systems combining speech and gestures, our system enables real‑time control of volume, brightness, media, applications, files, and AI chat—all running concurrently using multithreading for responsiveness. Evaluation demonstrates high accuracy and low latency, showing promise for intuitive, accessible multimodal interfaces.
Author: Pawan Gandhi, Krina Masharu
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7091612576
FWCI:
Cites:
Cited by:
Related to: