Progressively Optimized Local Radiance Fields for Robust View Synthesis

Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin Huang, Changil Kim, Min H. Kim,  and Johannes Kopf
IEEE/CFV Conference on Computer Vision and Pattern Recognition (CVPR), 2023

📝 Paper: https://arxiv.org/abs/2303.13791
🌐 Website: https://localrf.github.io/
💻 Code: https://github.com/facebookresearch/localrf
🗄️ Data: https://drive.google.com/drive/folders/1kGY-VijIbXNsNb7ghEywi1fvkH4BaIEz

📄 Abstract: We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.


🎵 Music by: Bensound.com/free-music-for-videos
License code: DSEOWWIQM8J3H4OZ

Jia-Bin Huang

A technique for creating immersive 3D renderings from casual, shaky videos by extending Neural Radiance Fields (NeRF). The approach introduces multiple local radiance fields along a camera trajectory to handle long sequences while maintaining quality. It also addresses the challenge of unreliable camera pose estimation from Structure from Motion (SfM) by progressively estimating poses within a local radiance field framework. Adjacent radiance fields are blended for seamless transitions, enabling high-quality free-view synthesis from unconstrained real-world video.

Immersive 3D Rendering from Casual Videos