Interpretable driver behavioral state monitoring using facial landmarks for human-autonomous vehicle interaction
Year: 2026
Type: article
Abstract: Introduction Behavior-aware robotic systems require both accurate human state recognition and transparent machine intent to support trust and safe collaboration. This study presents an integrated framework for autonomous vehicles combining a real-time vehicle state visualization interface with a calibration-driven facial-based behavioral state recognition module. Methods A Polaris GEM e6 autonomous vehicle equipped with drive-by-wire control and LiDAR served as the testbed. A real-time Human-Machine Interface (HMI) built with ROS and Foxglove Studio visualized localization, object detection, and driving states. Concurrently, a vision-based behavioral monitoring system used an Intel RealSense L515 camera, MediaPipe, and OpenCV to extract facial landmarks. Individual calibration established neutral baselines for eye aspect ratio, mouth aspect ration, and head rotation angles. A real-time threshold-based finite-state model facilitated data collection across five behavioral states: neutral, closed eyes, out-of-safe-zone, yawning, and fast blinking. Classical classifiers (Support Vector Machine [SVM], Random Forest [RF]), and deep learning architectures (Convolutional Neural Networks, Recurrent Neural Networks) were trained on a controlled indoor dataset using stratified k-fold cross-validation (k = 10), and evaluated on a hold-out outdoor dataset. Results The HMI provided stable, real-time visualization of vehicle state and perception, while the facial monitoring system achieved a ∼ 98.5% face detection success rate. The online heuristic model identified behavioral states and enabled high-quality data collection, capturing features with strong Kruskal-Wallis significance from the neutral state for closed eyes (EAR, p < 0.001 ), yawning (MAR, p < 0.001 ), fast blinking (blink rate, p < 0.01 ), though out-of-safe-zone was non-significant (head rotation, p > 0.05 ). Stratified cross-validation accuracy and F1-scores reached ∼ 91% for SVM and RF. On the outdoor test data, SVM maintained its performance, whereas RF dropped to a ∼ 70% F1-score due to inter-session variability and feature ambiguity. Deep learning architectures achieved test F1-scores ( ∼ 80%), comparable to the heuristic baseline model ( ∼ 81%). SVM was the most stable and best performing classifier, isolating its confusion to the out-of-safe-zone and neutral states. Conclusion Combining transparent interface design with personalized calibration supports human-aware autonomy. The machine learning framework provided interpretable monitoring, while real-time visualization enhanced situational awareness, together contributing to behaviorally responsive and transparent human-robot interaction platforms.
Source: Frontiers in Robotics and AI
Institution: Tecnológico de Monterrey
Language: English
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,353
OpenAlex ID: W7223259968