Joint alignment of synthetic and real domains for robotics

Released in: DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer



Generating large-scale synthetic data in simulation is a feasible alternative to collecting/labelling real data for training vision-based deep learning models, albeit the modelling inaccuracies do not generalize to the physical world. In this paper, we present a domain-invariant representation learning (DIRL) algorithm to adapt deep models to the physical environment with a small amount of real data. Existing approaches that only mitigate the covariate shift by aligning the marginal distributions across the domains and assume the conditional distributions to be domain-invariant can lead to ambiguous transfer in real scenarios. DIRL jointly aligns the marginal (input domains) and the conditional (output labels) distributions to mitigate the covariate and the conditional shift across the domains with adversarial learning, and combines it with a triplet distribution loss to make the conditional distributions disjoint in the shared feature space. Experiments on digit domains yield state-of-the-art performance on challenging benchmarks, while sim-to-real transfer of object recognition for vision-based decluttering with a mobile robot improves from 26.8 % to 91.0 %, resulting in 86.5 % grasping accuracy of a wide variety of objects.


Year Released

Key Links & Stats


DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer

ML Tasks

  1. Domain Adaptation
  2. Object Detection
  3. Object Recognition

ML Platform

  1. Tensorflow


  1. General
  2. Still Image
  3. Video


  1. General
  2. Industrial & Warehouse Robotics

CG Platform

  1. Not Applicable

Related organizations

UC Berkeley