Real-Time View Synthesis with Multiplane Image Network using Multimodal Supervision
Year: 2025
Type: conference-paper
Abstract: Recent advances in view synthesis from a single image have increased visual quality of the newly synthesized viewpoints significantly. However, the high computational cost of state-of-the-art methods remains a critical bottleneck, limiting their adoption in real-time applications such as immersive telepresence. To address this limitation, we present a multiplane image (MPI) network that achieves real-time view synthesis. Unlike existing approaches that often rely on a separate depth estimation network to guide the network for estimating MPI parameters, our framework directly predicts the MPI parameters from a single RGB image. To guide the network for estimating correct parameters, we introduce a training strategy that leverages joint supervision from both view synthesis and depth estimation losses to ensure visual fidelity. During inference, our method exclusively utilizes the optimized view synthesis branch, while the depth decoder is only used for training. Our end-to-end approach renders views from a single input image in real-time. Extensive experiments validate that our method provides a compelling rendering speed with visual quality in par with state-of-the-art methods, highlighting its suitability for live, interactive applications.
Language: English
Topic: Advanced Vision and Imaging
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Knowledge Foundation
OpenAlex ID: W7123525309