Single-Image 3D Human Digitization with Shape-Guided Diffusion

Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang

ACM SIGGRAPH Asia 2023

📝 Paper: https://human-sgd.github.io/
🌐 Website: https://human-sgd.github.io/
💻 Code: https://human-sgd.github.io/

Abstract: We present an approach to generate a 360-degree view of a person with a consistent, high-resolution appearance from a single input image. NeRF and its variants typically require videos or images from different viewpoints. Most existing approaches taking monocular input either rely on ground-truth 3D scans for supervision or lack 3D consistency. While recent 3D generative models show promise of 3D consistent human digitization, these approaches do not generalize well to diverse clothing appearances, and the results lack photorealism. Unlike existing work, we utilize high capacity 2D diffusion models pretrained for general image synthesis tasks as an appearance prior of clothed humans. To achieve better 3D consistency while retaining the input identity, we progressively synthesize multiple views of the human in the input image by inpainting missing regions with shape guided diffusion conditioned on silhouette and surface normal. We then fuse these synthesized multi-view images via inverse rendering to obtain a fully textured high-resolution 3D mesh of the given person. Experiments show that our approach outperforms prior methods and achieves photorealistic 360-degree synthesis of a wide range of clothed humans with complex textures from a single image.

Compared baseline methods:
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization, ICCV 2019
https://shunsukesaito.github.io/PIFu/
- Text-Guided Texturing of 3D Shapes, SIGGRAPH 2023
https://texturepaper.github.io/TEXTurePaper/
- Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors, arXiv 2023
https://guochengqian.github.io/project/magic123/
- TeCH: Text-guided Reconstruction of Lifelike Clothed Humans
https://huangyangyi.github.io/TeCH/

Jia-Bin Huang

A research method for reconstructing a detailed 3D model of a clothed human from a single image. The approach first predicts shape using PIFu-HD and synthesizes back-view appearance using Pose-with-Style, then iteratively generates multi-view images guided by 3D surface normals and contours using a diffusion model. The multi-view images are consolidated into a consistent UV texture map via differentiable rendering. Compared to prior methods like PIFu, TEXTure, Magic-123, and TeCH, this approach produces fewer artifacts and more consistent full-body textures, though limitations include baked-in lighting and inability to refine existing 3D shapes.

3D Human Digitization from a Single Image!