Year: 2025
Type: dissertation
Abstract: Human activity recognition (HAR) is a vital area of research in pattern recognition and computer vision. Its practical applications in daily life, particularly those aimed at improving quality of life, have attracted widespread attention. In healthcare, for example, HAR can be used to evaluate medical procedures such as endotracheal suctioning. This study focuses on three HAR tasks: sign language recognition (SLR), endotracheal suctioning (ES) recognition, and inner speech recognition using electroencephalogram (EEG) signals. The key objective is to make HAR systems more accessible and portable. Two main challenges arise in this context: (1) achieving real-time performance and (2) enabling low-cost computation. Many prior studies utilize deep learning or deep RNN; however, these approaches are not well-suited for portable devices due to their high computational demands. Conversely, traditional machine learning methods struggle with dynamic input, especially video-based data. Therefore, this study emphasizes reservoir computing (RC), particularly echo state network (ESN), due to their suitability for temporal data. However, there are some challenge that faced in this study because the standard ESN did not perform optimally to solve the complex HAR, such as: 1. Single input modality may be insufficient. 2. There is a lack of training data. 3. Low memory capacity in the standard ESN. 4. One reservoir faces difficulty in extracting various features in a complex task. 5. A single readout struggles to classify many patterns. In the sign language task, three datasets are used in this study: the WLASL100, WLASL300 (American sign language) dataset and the LSA64 (Argentinian sign language) dataset. SLR faces problems such as the difficulty of extracting various features, the inadequacy of one modality, the low memory capacity of the ESN, and the inadequacy of one readout to solve complex problems. Four architectures are proposed to solve these problems in SLR tasks: multiple reservoir computing to extract various features, multimodality to provide additional modalities to the reservoir model, deep ESN to increase memory capacity, and ensemble learning of deep ESN and multiple readouts. The ensemble learning method demonstrated the highest level of accuracy, achieving 76.12% for WLASL100, 58.20% for WLASL300, and 91.41% for signer independence on LSA64. ES activity is associated with complications and risks. Recognizing ES activity ensures patient safety and improves nurses’ skills when conducting complicated procedures. Data collection presents more challenges, including missing values. Using generative AI and LLM can help solve this issue by analyzing and generating data. This study uses a combination of LLM and multiple readouts to classify ES labels. The optimal parameters for this task were 16 grids and 210 nodes. The classification accuracy was 70.5%. In the inner speech recognition by using EEG signals task, there is a problem of a lack of modality. This study used bidirectional RC to improve the performance of unidirectional RC. The highest accuracy parameter in the EEG task is 18.94%. The dataset’s low accuracy is due to the need for additional preprocessing to reduce noise.
DOI: 10.18997/0002002144
Author: Arie Rachmad Syulistyo,
Institution: Kyushu Institute of Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7116114263
Cites:
Cited by:
Related to: