Replacing Labeled Real-Image Datasets with Auto-Generated Contours

Improved pretraining for vision transformers based on formula-generated synthetic fractal images

Released in: Replacing Labeled Real-Image Datasets with Auto-Generated Contours



The work shows that the performance of formula-driven supervised learning (FDSL) can match or even exceed that of ImageNet-21k without the use of real images, human-, and self-supervision during the pre-training of Vision Transformers (ViTs). For example, ViT-Base pre-trained on ImageNet-21k shows 81.8% top-1 accuracy when fine-tuned on ImageNet-1k and FDSL shows 82.7% top-1 accuracy when pre-trained under the same conditions (number of images, hyperparameters, and number of epochs). Images generated by formulas avoid the privacy/copyright issues, labeling cost and errors, and biases that real images suffer from, and thus have tremendous potential for pre-training general models. To understand the performance of the synthetic images, the authors tested two hypotheses, namely (i) object contours are what matter in FDSL datasets and (ii) increased number of parameters to create labels affects performance improvement in FDSL pre-training. To test the former hypothesis, we constructed a dataset that consisted of simple object contour combinations. We found that this dataset can match the performance of fractals. For the latter hypothesis, we found that increasing the difficulty of the pre-training task generally leads to better fine-tuning accuracy.


Year Released

Key Links & Stats

ExFractalDB and RCDB

Replacing Labeled Real-Image Datasets with Auto-Generated Contours

@InProceedings{Kataoka_2022_CVPR, author = {Kataoka, Hirokatsu and Hayamizu, Ryo and Yamada, Ryosuke and Nakashima, Kodai and Takashima, Sora and Zhang, Xinyu and Martinez-Noriega, Edgar Josafat and Inoue, Nakamasa and Yokota, Rio}, title = {Replacing Labeled Real-Image Datasets With Auto-Generated Contours}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2022}, pages = {21232-21241} }

ML Tasks

  1. General
  2. Image Classification

ML Platform

  1. Pytorch


  1. Still Image


  1. General

CG Platform

  1. Other

Related organizations

National Institute of Advanced Industrial Science and Technology (AIST)

Tokyo Institute of Technology