Detecting Learners' Hint Inquiry Behaviors in e-Learning Environment by using Facial Expressions
Year: 2023
Type: conference-paper
Abstract: It is difficult for teachers to track how each student is engaged in a class through e-learning. The current study simulated an e-learning environment by designing a website which contains interactable functions and hint buttons to examine whether the mental state of hint seeking can be estimated from facial features. We currently recruited 9 participants and asked them to solve a problem used in the International Olympiad of Linguistics in 2018. In this study, participants learned alone without discussing with others on a website designed for the experiment. While they were solving the problems, their face images were recorded. We extracted their facial features as Action Unit codes (AUs) by an open-source software, OpenFace. There was a function that provided hints to participants when he or she clicked a button. We attempted to estimate the time to click the button, that is the time when a hint was required, from facial expressions using a machine learning technique. We used LightGBM for the prediction. The model trained by LightGBM classified the state of "seeking hints" and the other states with an accuracy of 85.0%. The SHAP (SHapley Additive exPlanations) values, which are indexes of contribution levels, showed that facial features of "brow lowerer", "lip tightener", and "jaw drop" are the top three important features for the classification. This suggests that it is possible to build a system that detects the mental state of hint-seeking from facial features.
Authors: Guan-Yun Wang, Hikaru Nagata, Yasuhiro Hatori, Yoshiyuki Sato, Chia-Huei Tseng, Satoshi Shioiri
Institution: Tohoku University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
OpenAlex ID: W4384770596