Work

Robustness of multimodal large language models in facial expression recognition: A human-comparative study

Year: 2026

Type: article

Abstract: Multimodal Large Language Models (MLLMs) have achieved strong performance on standard facial expression recognition (FER) benchmarks. However, high accuracy on canonical expressions does not necessarily indicate robustness when facial information is conflicting or incomplete. This study investigates the alignment between MLLMs and human response patterns under two controlled challenges: semantic conflict and information scarcity. We introduce a bio-inspired dual-task evaluation framework comparing human observers with state-of-the-art (SOTA) MLLMs. First, utilizing the Composite Face paradigm, we examine model robustness against conflicting emotional cues to test susceptibility to semantic interference. Second, employing ocular stimuli, we evaluate the models’ capacity to infer basic emotions from isolated local features (isolated eyes, eyes and eyebrows) versus the full facial information. MLLMs showed higher accuracy for incongruent than congruent composite faces, opposite to the human pattern, and their accuracy declined more sharply than human accuracy as visible facial information was restricted. The main regional pattern was reproduced under matched dynamic presentation and alternative prompt templates, although its exact magnitude varied across models and prompts. Humans also showed lower group-level calibration error than the evaluated MLLMs. We propose this framework as a benchmark-style evaluation suite for stress-testing perceptual robustness and cognitive alignment in next-generation affective computing systems.

Source: Pattern Recognition

DOI: 10.1016/j.patcog.2026.115177