Motion-Dependent Appearance for Dynamic Humans From a Single Camera

Learning to model secondary motion, e.g., clothes for a danching human, via equivariant transformations

Released in: Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans From a Single Camera



Appearance of dressed humans undergoes a complex geometric transformation induced not only by the static pose but also by its dynamics, i.e., there exists a number of cloth geometric configurations given a pose depending on the way it has moved. Such appearance modeling conditioned on motion has been largely neglected in existing human rendering methods, resulting in rendering of physically implausible motion. A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations. This paper presents a compact motion representation by enforcing equivariance — a representation is expected to be transformed in the way that the pose is transformed. The authors model an equivariant encoder that can generate the generalizable representation from the spatial and temporal derivatives of the 3D body surface. This learned representation is decoded by a compositional multi-task decoder that renders high fidelity time-varying appearance. Experiments show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.


Year Released

Key Links & Stats

@InProceedings{Yoon_2022_CVPR, author = {Yoon, Jae Shin and Ceylan, Duygu and Wang, Tuanfeng Y. and Lu, Jingwan and Yang, Jimei and Shu, Zhixin and Park, Hyun Soo}, title = {Learning Motion-Dependent Appearance for High-Fidelity Rendering of Dynamic Humans From a Single Camera}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {3407-3417} }

ML Tasks

  1. General

ML Platform

  1. Not Applicable


  1. 3D Asset


  1. Digital Human

CG Platform

  1. Not Applicable

Related organizations

Adobe Research

University of Minnesota