Videos are windows to another world. But the videos today are *flat*, confined to the original viewpoints. We showcase a method for converting any 2D videos into 3D videos that allow free-view synthesis.

Fast View Synthesis of Casual Videos
Yao-Chih Lee, Zhoutong Zhang, Kevin Blackburn-Matzen, Simon Niklaus, Jianming Zhang, Jia-Bin Huang, and Feng Liu
European Conference on Computer Vision, 2024

📝 Paper: https://arxiv.org/abs/2312.02135
🌐 Website: https://casual-fvs.github.io/

Abstract: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

Transcript: Available in the subtitle.

Credit: The "versus" icon is from PNGtree.

Jia-Bin Huang

A research method for converting any 2D video into an immersive 3D experience that supports changing camera viewpoints, focal lengths, and space-time view synthesis. The approach models static scene components as collections of 3D planes with view-dependent textures, and dynamic content as deformable 3D point clouds rendered via forward splatting. It addresses the core challenge of limited viewpoint variation in casual videos, outperforming methods like 4D Gaussian Splatting and DALL in visual quality while being up to 600x faster to render. Known limitations include motion segmentation failures and distortion artifacts at viewpoints far from the original trajectory.

How We Can Convert Any Videos to 3D