Multimodal confusion state detection in Japanese text reading using Vision Transformers, facial video, and eye tracking
Year: 2025
Type: article
Abstract: The COVID-19 pandemic has led to a significant increase in online classes, necessitating robust support system for students in virtual learning environments. One critical aspect of such support system is the detection of confusion during text-based learning, as students may struggle with comprehension. In this study, we propose a multimodal data fusion approach leveraging machine learning models to detect confusion states in Japanese text reading. The proposed method integrates facial video and eye-tracking data using a Vision Transformer-based video classification model. The novelty of our approach lies in the early-stage data fusion, performed prior to model input, where facial images are augmented with eye-tracking features. Experimental results demonstrate that this fusion strategy, achieved through facial image overwriting, enhances confusion state detection accuracy compared to conventional late fusion machine learning models. These findings highlight the potential of the proposed Vision Transformer-based method as a simple yet effective framework for multimodal data analysis using image-based overwriting.
Source: Displays
Authors: Yudai Ito, Chikako Ishizawa, Yoichi Kageyama
Institution: Akita University
Language: English
Subfield: Human-Computer Interaction
Field: Computer Science
Domain: Physical Sciences
SDG: Quality education
Open Access status: hybrid
Estimated APC paid: $2,450
OpenAlex ID: W4415559285