AI-Based Indian Sign Language Recognition, Real-Time Speech Generation Using Deep Learning and Computer Vision
Year: 2026
Type: article
Abstract: Communication between people who use Indian Sign Language (ISL) and people who do not understand it can be challenging. This project presents a camera-based ISL recognition system that converts hand gestures into letters and words, supports rule-based sentence formation, and converts selected sentences into speech. OpenCV processes camera frames, while MediaPipe Hands extracts hand landmarks. The letter-recognition module uses 21 landmarks with X, Y, and Z coordinates, producing 63 features that are classified using a Random Forest model. The dataset contains 30,884 samples, and the reported accuracy is approximately 99.97%. The word-recognition module extracts 162 features per frame and processes 16-frame sequences using a TensorFlow/Keras architecture with Conv1D, Multi-Head Attention, and Transformer-based components. It recognizes 23 predefined words and achieves 87.67% accuracy. Recognized words are stored and passed to a rule-based sentence-formation module, where the user selects a sentence for Text-to-Speech using pyttsx3. An integrated Speech-to-Text module converts spoken input into text, supporting communication in the reverse direction. A Tkinter application provides learning and recognition modes for letters and words, sentence recognition, and Speech-to-Text.
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225657474
FWCI:
Cites:
Cited by:
Related to: