Real-Time Multimodal Assistive Communication Using Rule-Based Hand Gesture Recognition and Offline Speech-to-Text Conversion
Year: 2026
Type: conference-paper
Abstract: Simple everyday encounters can frequently become major obstacles for those with speech or hearing impairments, especially in settings without specialised assistive infrastructure. In order to close this communication gap, this research presents a real-time, lightweight multimodal system that combines a completely offline speech-to-text engine with camera-based hand gesture detection. We convert physical gestures into meaningful vocal words without requiring the large datasets that machine learning models usually demand by using Media Pipe to track key hand landmarks and a simplified rule-based encoding technique. In order to process spoken language locally and maintain functioning even in the absence of an internet connection, the system simultaneously uses the Vosk and Kaldi engines. According to our research, this integrated approach offers nearly immediate response times with low latency, providing a useful and approachable tool for improving assistive communication in a variety of real-world contexts.
Institution: Yeshwantrao Chavan College of Engineering
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7213367196