3D reconstruction with deformable animations from casual video collections

Released in: BANMo: Building Animatable 3D Neural Models from Many Casual Videos



Prior work for articulated 3D shape reconstruction often relies on specialized sensors (e.g., synchronized multi-camera systems), or pre-built 3D deformable models (e.g., SMAL or SMPL). Such methods are not able to scale to diverse sets of objects in the wild. This work present BANMo, a method that requires neither a specialized sensor nor a pre-defined template shape. BANMo builds high-fidelity, articulated 3D models (including shape and animatable skinning weights) from many monocular casual videos in a differentiable rendering framework. While the use of many videos provides more coverage of camera views and object articulations, they introduce significant challenges in establishing correspondence across scenes with different backgrounds, illumination conditions, etc. The key insight here is to merge three schools of thought; (1) classic deformable shape models that make use of articulated bones and blend skinning, (2) volumetric neural radiance fields (NeRFs) that are amenable to gradient-based optimization, and (3) canonical embeddings that generate correspondences between pixels and an articulated model. The authors introduce neural blend skinning models that allow for differentiable and invertible articulated deformations. When combined with canonical embeddings, such models allow to establish dense correspondences across videos that can be self-supervised with cycle consistency. On real and synthetic datasets, BANMo shows higher-fidelity 3D reconstructions than prior works for humans and animals, with the ability to render realistic images from novel viewpoints and poses.


Year Released

Key Links & Stats

BANMo: Building Animatable 3D Neural Models from Many Casual Videos

@InProceedings{Yang_2022_CVPR, author = {Yang, Gengshan and Vo, Minh and Neverova, Natalia and Ramanan, Deva and Vedaldi, Andrea and Joo, Hanbyul}, title = {BANMo: Building Animatable 3D Neural Models From Many Casual Videos}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {2863-2873} }

ML Tasks

  1. NERF

ML Platform

  1. Not Applicable


  1. Video
  2. 3D Asset


  1. Digital Human
  2. Synthetic Media & Art

CG Platform

  1. Not Applicable

Related organizations

Meta AI

Carnegie Mellon University

Meta Reality Labs