Controlled Evaluation of Gated Multimodal Fusion for Lie Detection Using Facial Motion and Speech Cues
Year: 2026
Type: article
Abstract: Automatic lie detection from video requires combining weak behavioral cues under clip-level supervision. In gated multimodal models, the effects of modality-prior initialization and continuing regularization are difficult to distinguish without matched controls. This study evaluates a three-stream DOLOS pipeline combining static face crops, short-term optical flow, Wav2Vec2-base speech representations, and face-validity masks. Three gated logit-fusion conditions are compared: uniform initialization without regularization, directed initialization without regularization, and directed initialization with Prior-KL. Each condition is evaluated across three predefined splits and three random seeds. The directed-initialized Prior-KL model achieves 64.16 ± 4.56% AUC, 58.96 ± 4.16% balanced accuracy at the validation-selected threshold, 56.74 ± 10.67% lie-class F1, and 57.45 ± 5.15% macro-F1. Relative to the uniform-initialized control, the combined initialization-plus-KL configuration gains 2.26 percentage points in AUC and 2.29 points in macro-F1. With directed initialization held constant, adding KL yields higher lie-class F1 but no improvement in AUC or balanced accuracy. A matched seed-42 ablation finds that static facial appearance alone achieves higher AUC and balanced accuracy than the fused model, while fusion yields higher lie-class F1. These findings associate the observed mean AUC gain with directed initialization and show a class-specific trade-off from regularization. They also motivate identity-disjoint evaluation to assess the role of appearance-related shortcuts.
Author: Huy Phan Anh
Institution: Saigon University
Language: English
Subfield: Social Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7225449961