Portrait Eyeglasses and Shadow Removal by Leveraging 3D Synthetic Data

New synthetic dataset used to train a state of the art model for eyeglass removal

Released in: Portrait Eyeglasses and Shadow Removal by Leveraging 3D Synthetic Data

Contributor:

Summary

In portraits, eyeglasses may occlude facial regions and generate cast shadows on faces, which degrades the performance of many techniques like face verification and expression recognition. Portrait eyeglasses removal is critical in handling these problems. However, completely removing the eyeglasses is challenging because the lighting effects (e.g., cast shadows) caused by them are often complex. In this paper, the authors propose a novel framework to remove eyeglasses as well as their cast shadows from face images. The method works in a detect-then-remove manner, in which eyeglasses and cast shadows are both detected and then removed from images. Due to the lack of paired data for supervised training, the paper presents a new synthetic portrait dataset with both intermediate and final supervisions for both the detection and removal tasks. Furthermore, the authors apply a cross-domain technique to fill the gap between the synthetic and real data. The proposed technique is claimed to be the first to remove eyeglasses and their cast shadows simultaneously.

2022

Year Released

Key Links & Stats

take-off-eyeglasses

Portrait Eyeglasses and Shadow Removal by Leveraging 3D Synthetic Data

@inproceedings{lyu2022portrait, title={Portrait Eyeglasses and Shadow Removal by Leveraging 3D Synthetic Data}, author={Lyu, Junfeng and Wang, Zhibo and Xu, Feng}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={3429--3439}, year={2022} }

ML Tasks

  1. Facial Modeling
  2. Image Generation

ML Platform

  1. Pytorch

Modalities

  1. Still Image

Verticals

  1. Digital Human

CG Platform

  1. Not Applicable

Related organizations

Tsinghua University