Year: 2024
Type: preprint
Abstract: Integrating and processing information from various sources or modalities are critical for obtaining a comprehensive and accurate perception of the real world in autonomous systems and cyber-physical systems. Drawing inspiration from neuroscience, we develop the Information-Theoretic Hierarchical Perception (ITHP) model, which utilizes the concept of information bottleneck. Different from most traditional fusion models that incorporate all modalities identically in neural networks, our model designates a prime modality and regards the remaining modalities as detectors in the information pathway, serving to distill the flow of information. Our proposed perception model focuses on constructing an effective and compact information flow by achieving a balance between the minimization of mutual information between the latent state and the input modal state, and the maximization of mutual information between the latent states and the remaining modal states. This approach leads to compact latent state representations that retain relevant information while minimizing redundancy, thereby substantially enhancing the performance of multimodal representation learning. Experimental evaluations on the MUStARD, CMU-MOSI, and CMU-MOSEI datasets demonstrate that our model consistently distills crucial information in multimodal learning scenarios, outperforming state-of-the-art benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-level performance in the multimodal sentiment binary classification task across all evaluation metrics (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and Pearson Correlation).
Source: arXiv (Cornell University)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Embracing Complexity: A Fractal Calculus Approach to the Modeling and Optimization of Medical Cyber-Physical Systems, FMitF: A Novel Framework for Learning Formal Abstractions and Causal Relations from Temporal Behaviors, CENTER FOR COMPLEX PARTICLE SYSTEMS (COMPASS), CPS: Small: Uncertainty-aware Framework for Specifying, Designing and Verifying Cyber-Physical Systems, G6193981907, Collaborative Research: MODULUS: A Novel Spatiotemporal Multifractal Analysis to Evaluate Genome Dynamics, G7452299184
OpenAlex ID: W4394866555