Work

Multi-Dimension-Embedding-Aware Modality Fusion Transformer for Psychiatric Disorder Diagnosis

Year: 2024

Type: conference-paper

Abstract: Deep learning and neuroimaging techniques are important in diagnosing psychiatric disorders. However, previous studies on psychiatric disorders classification mainly focus on using functional connectivity matrices of resting-state functional magnetic resonance imaging(rs-fMRI) as input. This approach fails to fully utilize the rich temporal information present in rs-fMRI data. Additionally, deep learning models equipped with effective modality fusion strategies have the potential to better utilize multi-modality data, thereby enhancing model performance. In this work, we proposed a multi-dimension-embedding-aware modality fusion transformer (MFFormer) for diagnosing schizophrenia and bipolar disorder using rs-fMRI and T1 weighted structural MRI (T1w sMRI). Concretely, to fully utilize the temporal information of rs-fMRI and spatial information of sMRI, we constructed a deep learning architecture that takes as input 2D time series of rs-fMRI and 3D volumes T1w. Furthermore, to promote intra-modality attention and information fusion across different modalities, a fusion transformer module (FTM) is designed through extensive self-attention of hybrid feature maps of multi-modality. In addition, a dimension-up and dimension-down strategy is suggested to properly align feature maps of multi-dimensional from different modalities. Experimental results on our private and public OpenfMRI datasets show that our proposed MFFormer performs better than that using a single modality or multi-modality MRI on schizophrenia and bipolar disorder classification.

DOI: 10.1145/3718441.3718466