PHORHUM

Monocular reconstruction in PIFu style with disentangled colors

Released in: Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing

Contributor:

Summary

The authors present PHORHUM, a novel, end-to-end trainable, deep neural network methodology for photorealistic 3D human reconstruction given just a monocular RGB image. The proposed pixel-aligned method estimates detailed 3D geometry and, for the first time, the unshaded surface color together with the scene illumination. Observing that 3D supervision alone is not sufficient for high fidelity color reconstruction, the authors introduce patch-based rendering losses that enable reliable color reconstruction on visible parts of the human, and detailed and plausible color estimation for the non-visible parts. Moreover, the method specifically addresses methodological and practical limitations of prior work in terms of representing geometry, albedo, and illumination effects, in an end-to-end model where factors can be effectively disentangled. In extensive experiments, the authors demonstrate the versatility and robustness of their approach; state-of-the-art results validate the method qualitatively and for different metrics, for both geometric and color reconstruction.

2022

Year Released

Key Links & Stats

Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing

@inproceedings{alldieck2022phorhum, title = {Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing}, author = {Thiemo Alldieck and Mihai Zanfir and Cristian Sminchisescu}, year = {2022}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)} }

ML Tasks

  1. Facial Modeling
  2. Human Pose Estimation

ML Platform

  1. Not Applicable

Modalities

  1. 3D Asset

Verticals

  1. Facial
  2. Digital Human

CG Platform

  1. Not Applicable

Related organizations

Google Research