Space-time video super-resolution based on temporal feature refinement network
Year: 2024
Type: article
Abstract: Space-time video super-resolution( STVSR) enhances video quality across both temporal and spatial dimensions, enabling real-time presentation of high-resolution and high-frame-rate videos despite limitations in video capture devices, transmission, or storage, thus meeting the demand for ultra-high-definition image quality. Compared to two-stage methods, one-stage approaches achieve frame interpolation at the feature level rather than the pixel level,significantly outperforming in terms of inference speed and computational complexity. Some existing one-stage STVSR methods employ pixel hallucination-based feature interpolation, which struggles to predict fast-moving objects between frames. To address this, a pyramid encoder-decoder network based on optical flow for temporal feature interpolation is proposed, to achieve rapid bidirectional optical flow estimation and more realistic texture synthesis. This network structure, termed temporal feature refinement network(TFRnet), enhances efficiency while mitigating the instability of optical flow estimation for large motions. Additionally, the spatial module incorporates sliding window-based local propagation and bidirectional propagation based on recurrent networks to strengthen frame alignment. To further exploit TFRnet′s potential, spatial super-resolution is prioritized over temporal super-resolution(space-first approach).Experiments on several widely used data benchmarks and evaluation metrics demonstrate the excellent performance of our proposed method, TFRnet-sf. While improving overall peak signal-to-noise ratio(PSNR) and structural similarity index(SSIM), it also enhances PSNR and SSIM for inserted intermediate frames, alleviating to some extent the issue of significant disparities in PSNR and SSIM between inserted intermediate frames and original frames.
Author: YAO Xiaojuan; MU Ke; PAN Pei; YANG Ziyi; ZHAO Yufei; ZHU Yonggui
Institution: Communication University of China
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7084148503
FWCI:
Cites:
Cited by:
Related to: