Reconstructing resolution-independent materials represented by procedural graphs from a coarse 3D model and a photo of an interior

Released in: PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes



Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. This work goes beyond this to propose PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. PhotoScene models scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. The model optimizes the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. The authors evaluate their technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that the proposed method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.


Year Released

Key Links & Stats

@InProceedings{Yeh_2022_CVPR, author = {Yeh, Yu-Ying and Li, Zhengqin and Hold-Geoffroy, Yannick and Zhu, Rui and Xu, Zexiang and Ha\v{s}an, Milo\v{s} and Sunkavalli, Kalyan and Chandraker, Manmohan}, title = {PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {18562-18571} }

ML Tasks

  1. General

ML Platform

  1. Not Applicable


  1. 3D Asset


  1. Industrial & Warehouse Robotics
  2. Synthetic Media & Art
  3. Home/Office

CG Platform

  1. Not Applicable

Related organizations

University of California, San Diego

Adobe Research