Work

Recognizing Indonesian words based on visual cues of lip movement using deep learning

Year: 2025

Type: article

Abstract: Lipreading is one of the techniques that can enhance speech perception. However, there are still limited studies of lipreading research focusing on low-resource languages, such as Indonesian. In this study, we introduce an instrument designed to generate lipreading datasets using CC BY video data available on YouTube called Lipreading Information Resource Assembler-Generator (LIRA-Gen). Using this instrument, we present the first Indonesian language lipreading dataset (IDLRW) containing over 48,000 videos with 100-word categories spoken by various persons in natural conditions. Also, we developed a deep learning architecture consisting of an Advanced Residual Network (ARN) using ResNet-34 incorporated with a Channel Spatial Attention (CSA) module, improved sequence modeling by fusing Bi-Gru with Mamba (BGM), an integrated word decision module, and fine-tuned hyperparameter. Our measurement shows that it reaches an accuracy of 60.51% on the IDLRW dataset and outperforms state-of-the-art lipreading models from another dataset even without implementing an additional learning strategy.

Source: Measurement

DOI: 10.1016/j.measurement.2025.116968

Authors: Griffani Megiyanto Rahmatullah, Shanq-Jang Ruan, Lieber Po-Hung Li