Multimodal Sensing and Machine Learning to Compare Printed and Verbal Assembly Instructions Delivered by a Social Robot
Year: 2025
Type: conference-paper
Abstract: In this paper, we compare a manual assembly task communicated to workers using both printed and robot-delivered instructions. The comparison was made using physiological signals (blood volume pulse (BVP) and electrodermal activity (EDA)) collected from individuals during an experimental study. In addition, we also collected responses using the NASA Task Load Index (TLX) survey. Furthermore, we mapped the collected physiological signals to the responses of participants for NASA TLX to predict their workload. For both classification problems, we compared the performance of Convolutional Neural Networks (CNNs) and Long-Short-Term Memory (LSTM) models. Results show that for our CNN-based approach using multimodal data including both BVP and EDA gave better results than using just BVP (approx. 8.38% better) and EDA (approx. 20.49% better). Furthermore, our LSTM-based approach also had better results when we used multimodal data (approx. 8.38% better than just BVP and 6.70% better than just EDA). Overall, CNNs performed 7.72% better than LSTMs for classifying physiologies for paper vs robot-based instruction. The CNN-based model also provided better classification results (approx. 17.83% better on an average across all responses of the NASA TLX) within a few minutes of training compared to the LSTM-based models.
Institution: University of Louisville
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
Funder: National Science Foundation
OpenAlex ID: W4414432401