Introducing TextureDreamer! TextureDreamer transfers textures from a few images to arbitrary 3D shapes. Excited about democratizing 3D content creation!

TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion
Yu-Ying Yeh, Jia-Bin Huang, Changil Kim,  Lei Xiao, Thu Nguyen-Phuoc, Numair Khan, Cheng Zhang, Manmohan Chandraker, Carl S Marshall, Zhao Dong, and Zhengqin Li
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2024

📝 Paper: https://arxiv.org/abs/2401.09416
🌐 Website: https://texturedreamer.github.io/

Abstract: We present TextureDreamer, a novel image-guided texture synthesis method to transfer relightable textures from a small number of input images (3 to 5) to target 3D shapes across arbitrary categories.

Texture creation is a pivotal challenge in vision and graphics. Industrial companies hire experienced artists to manually craft textures for 3D assets. Classical methods require densely sampled views and accurately aligned geometry, while learning-based methods are confined to category-specific shapes within the dataset. In contrast, TextureDreamer can transfer highly detailed, intricate textures from real-world environments to arbitrary objects with only a few casually captured images, potentially significantly democratizing texture creation.

Our core idea, personalized geometry-aware score distillation (PGSD), draws inspiration from recent advancements in diffuse models, including personalized modeling for texture information extraction, variational score distillation for detailed appearance synthesis, and explicit geometry guidance with ControlNet. Our integration and several essential modifications substantially improve the texture quality. Experiments on real images spanning different categories show that TextureDreamer can successfully transfer highly realistic, semantically meaningful texture to arbitrary objects, surpassing the visual quality of previous state-of-the-art.

Transcript: Available in the subtitle.

Jia-Bin Huang

A method for transferring textures from input photos to 3D meshes is presented, lowering the barrier for 3D content creation. The approach learns a BRDF field (diffuse, roughness, metallic parameters) using neural implicit representations. It builds on text-to-3D diffusion techniques, fine-tuning a diffusion model with specific input images and a unique object identifier. Variational score distillation improves color quality, while ControlNet with surface normals as conditions adds geometric awareness to eliminate artifacts. The result is relatable, PBR-compatible texture maps that outperform prior methods like Latent-Paint and TEXTure.

3D Texture Made Easy