Breaking The Communication Barrier: Leveraging Xai And Imaging For Deaf Gesture Recognition
Year: 2024
Type: article
Abstract: Ignorance of the wider population in sign language severely limits the possibilities of communication between the people who are hearing-impaired or are deaf and those who have normal hearing while consequentially influencing the deaf community negatively. This paper dwells on the deep learning-based approach for the understanding of sign language (SL) gestures, to provide a way through the communication problem between hearing and deaf individuals. We performed the work based on two datasets, including the newly introduced Sign Language MNIST dataset, which is patterned after the classic MNIST format but focused on American Sign Language (ASL) gestures. The Sign Language MNIST dataset offers a multi-class problem with 24 letter classes, excluding J and Z, and was created using an image pipeline to expand the original hand gesture images with diverse modifications. We utilized a properly created Convolutional Neural Network (CNN) model to classify different types of hand motions in sign language. The second dataset used in this paper is also sign language consisting of various ASL movements. The approach we developed with the customized CNN model can be concluded as an excellent tool for recognizing the hand gestures of the deaf community, as it exhibited a 94% accuracy, 95% precision, 94% recall, and 94% F1 – score for the dataset of ASL gestures, which represents a clear improvement over the already existing technologies. For the dataset of multi-class, we achieved an accuracy of 96% and the results are illustrated in table 1. In the same way, we implemented Explainable AI (XAI) techniques which created Density Heatmaps that gave an extent to the interpretability of the CNN Model's decision-making process for hand gestures. Heatmaps that were produced add a lot of credibility to the features that were done as a whole aid to recognizing sign gestures. Through a vivid understanding of the results of our research, it is evident that the approach we selected is very effective in accurately representing sign gestures. It boasted an impressive track record, setting the benchmark with high percentages of both accuracy and performance. Future work includes enhancing the model performance by incorporating more diverse datasets, implementing advanced preprocessing techniques, and integrating other XAI methods for deeper insight into the model. This study contributes to the emerging field of SL recognition, offering an effective model for gesture recognition and insight into its interpretability.
Author: Rameesha Zia
Institution: Quaid-i-Azam University
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7127050948