On-the-fly Feature Aggregation for Real-time Free-viewpoint Dynamic Human Video from Sparse Views
Year: 2026
Type: article
Abstract: Generating free-viewpoint videos of dynamic humans is a challenging yet crucial problem with numerous applications in immersive telepresence, film production, and sports analytics. Existing methods either require dense camera setups or struggle with high computation costs and temporal incoherence. In this paper, we present FlyHuman, a novel approach that achieves real-time, temporal consistent free-viewpoint rendering of dynamic humans from as few as three input views, and even from a single view. We model the dynamic human body using deformable 3D Gaussian Splatting, enabling efficient rendering. As the videos stream in, we track the motion and update the color of each 3D Gaussian frame-by-frame. To ensure temporally consistent geometry, we propose a graph-based deformation method that deforms the 3D Gaussians on-the-fly. For accurate appearance modeling with insufficient observations from sparse views, we introduce a spatial-temporal feature aggregation (STFA) module that gradually refines the features of the Gaussians. Extensive experiments demonstrate that our approach outperforms previous work in terms of rendering quality, temporal consistency, and efficiency. Code and models will be publicly available.
Source: Fundamental Research
Authors: Wenbin Lin, Ruchang Yao, Jiaqi Li, Jun-Hai Yong, Feng Xu
Institution: Tsinghua University
Language: English
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7202359844