What does a multimodal EEG–Eye-tracking engagement decoder learn? A leakage-aware audit on the NeuMa dataset
Year: 2026
Type: article
Abstract: Multimodal decoders combining electroencephalography (EEG) and eye tracking predict consumer engagement, but accuracy against a rule-defined label does not establish which information a decoder uses. We audit this question on NeuMa (42 participants; 37 evaluable engagement folds), using an engagement index constructed by the authors, the Interpretable Neuro-Symbolic Hybrid Dynamic Graph Spiking Cross-Modal Transformer (INS-HDGS-CMT) as the architecture under audit, and EEGNet, ShallowConvNet and an eye-tracking long short-term memory network (LSTM) as references, under leave-one-subject-out cross-validation. Four findings emerge. First, gaze dominates predictability of the constructed index: a probe on the index’s five gaze terms reaches an area under the receiver operating characteristic curve (ROC-AUC) of 0.92, the gaze-only LSTM 0.89, and INS-HDGS-CMT 0.88; no multimodal model in the principal comparison exceeded the LSTM’s ROC-AUC. Second, in the principal tuned comparison, EEG encoders score below the EEG-term linear probe (0.49–0.64 versus 0.67), although EEGNet and ShallowConvNet distinguish resting state from browsing at 0.92–0.94; selected models retested under fold-wise relabelling preserved the broad EEG–gaze separation. Third, performance alone did not substantiate the decoder’s connectivity, efficiency or interpretability mechanisms: retrained density-matched topology surrogates produced no detectable loss, the spiking front-end was slower than a dense twin, and the bypass classifier dominated the final decision. Fourth, retrospective product-selection analyses yielded mean fold ROC-AUCs of 0.88–0.91 for session-level dwell probes, which include post-selection viewing, versus at most 0.61 for five-second epoch models. The contribution is these findings, together with an eight-step audit protocol for examining label dependence and component contributions, released with the code.
Source: Brain Informatics
Authors: D. Priyadharshini, S. Shridevi
Institution: Vellore Institute of Technology University
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: gold
Estimated APC paid: $1,090
OpenAlex ID: W7225404392