Work

Learning with Missing Data: A Multimodal Hierarchical Variational Auto-Encoders for Medical Image Synthesis

Year: 2026

Type: conference-paper

Abstract: Missing data is a widespread problem in multimodal medical imaging that presents significant practical and methodological challenges. While cross-modal synthesis has emerged as a promising strategy to estimate unavailable modalities from observed ones, most existing unified approaches assume complete multimodal datasets during training and rely on heuristic fusion strategies. In this short paper, we highlight the main ideas and findings of our recent work, where we introduced a Mixture of Multimodal Hierarchical Variational Auto-Encoders (MMHVAE) for unified cross-modal medical image synthesis from incomplete data. The model combines a hierarchical latent representation with a mixture of product-of-experts posterior to encode observed information, estimate missing information, and fuse arbitrary subsets of available modalities. The method was validated on the challenging problem of synthesis between multi-parametric MRI and intraoperative ultrasound in brain tumor patients, with additional evaluation on two downstream tasks: segmentation and registration. This work shows that principled probabilistic models can learn rich and informative multimodal representations from incomplete imaging datasets.

Source: HAL

Authors: Dorent, Reuben, Haouchine, Nazim, Golby, Alexandra, Frisken, Sarah, Kapur, Tina, Wells, Sandy

Institutions: Pitié-Salpêtrière Hospital, Institut du Cerveau, ARAMIS: Algorithmes, modèles et méthodes pour les images et les signaux du cerveau humain sain et pathologique, Harvard University

Language: English

Topic: Generative Adversarial Networks and Image Synthesis

Subfield: Computer Vision and Pattern Recognition

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7204218149

FWCI:

Cites:

Cited by:

Related to: