Fusing multimodal neural networks: a study on sleep classification and sound event localization and detection
Year: 2024
Type: article
Abstract: This is an explorative study of multimodal large-scale transformer networks.The thesis explores methods to pretrain and fuse multiple large-scale transformer networks each responsible for a modality, in order to improve their performance on different tasks.Specifically, we first dive into the task of infant sleep classification using audio, electrocardiogram (ECG), and inertial measurement unit (IMU).We explore various pretraining and finetuning schemes, as well as different fusion techniques.We also assess the effectiveness of fusion by cross-attention with sound event localization and detection (SELD), a multichannel machine learning task with multiple outputs.We show that this multimodal network structure is generic enough to work in various settings.
Author: Chang, Kai Chieh
Institution: University of Illinois Urbana-Champaign
Language: English
Topic: Music and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7220772532
