Using CycleGAN and Google’s QuickDraw dataset to create festive dragons and oxen

Friday, February 12th marks the start of the Lunar New Year in 2021. You’ll often see colorful dancing dragons at new year festivals wishing celebrators good luck. To celebrate ML-style, I used CycleGAN to generate dragons with fun decorative flare (shown above) and I’ll describe the implementation steps in this article. Perhaps these GAN-generated dragons will also bring luck! In Cantonese, we say "Gung Hay Fat Choy" to wish you happiness and prosperity in the new lunar year. I never miss an opportunity to use cheesy puns, so GAN Hay Fat Choy! 🙈
The Cyclegan paper was written by researchers from Berkeley AI Research and they have kindly released their code as well, making the implementation of this model very accessible. Their model enables unpaired image-to-image translation using cycle-consistent adversarial networks. Prior to this work, image-to-image translation required large datasets of paired images, before and after the desired image translation is applied. For example, to create summer to winter photo translations, you would have needed a large dataset of the exact same images in the winter and summer. With CycleGAN, you just need a dataset A of summer landscapes and a dataset B of winter landscapes and the model will learn the image translation – pretty impressive! Please refer to the paper for details.
Implementing CycleGAN

To test out CycleGAN, all you’ll need is a dataset A and dataset B of images along with some high computing power. My goal for this project was to create festive dragons with loud splashes of red, gold, and green like this dragon shown here. So, I needed a dataset A of dragons and a dataset B of colorful, loud flair – or in this case, fiery flare.
According to the authors, a dataset between 200 and 1000 images should be sufficient for CycleGAN to train on, though the more the merrier if you don’t mind long training times. For the dragons, I used Google’s QuickDraw dataset which consists of crowdsourced sketches from a drawing game they host. Given a word, the player must draw it in 20 seconds. They’ve released all the resulting drawings as an open dataset. I am a huge fan of QuickDraw as it allows us to peer into the world’s collective visual representations of different concepts. The time constraint forces us to focus on including only the most important details that visually describe the given word, allowing time for only the essence of a concept to be drawn. Looking through many of the dragon drawings, I consistently saw wings, a reptilian tail, and fire-breathing – the essence of a dragon, I suppose. I was very impressed by some of the dragon drawings completed in just 20 seconds. For my data, I collected about 450 dragon sketches for dataset A. Then, for dataset B, I created a dataset of 450 random shapes and fills – stochastic flare if you will. I illustrated how CycleGAN uses my datasets at a high level to learn image translation in the figure below.

Once you clone the authors’ Github, you’ll need to create a folder under datasets. In that folder, create "trainA", "trainB", "testA" and "testB" folders to house your data. Once your data is in place, you can run the following line of code, modifying the dataset location and model name as needed to start training the model.
python train.py --dataroot ./datasets/sketch2festivedragon --name cycle_gan_dragon_rg --model cycle_gan
On an NVIDIA GeForce RTX 3090, using a batch size of 1 (which is the default), the model was using about 10GB of GPU memory and training (for 200 epochs) completed in a little over 4 hours. If you use the authors’ code, I highly recommend following their training tips and running the following command to launch real-time training visualization on a local graphics web server.
python -m visdom.server

Training GANs is notoriously difficult because the loss plots are difficult to interpret and do not necessarily align with desired results. Instead, it is very important to visualize results in order to track progress towards your desired image translation. If you’re testing often, you may even see many instances of nice results across several different epochs. By default, the implementation code saves the model’s weights every 5 epochs. You can go back and test the model’s output using any of these checkpoint weights after training. All you have to do is run the following line of code and modify the dataset location, name of your model, and the epoch number. Test images will be saved in a results folder.
python test.py --dataroot ./datasets/sketch2festivedragon --name cycle_gan_dragon_rg --model cycle_gan --epoch 35
Looking at the test outputs from each epoch, I found lovely decorative dragons all across training. As I was looking for artistic flare, it was not too important that the dragons perfectly resembled the flare dataset. Towards the end of training, the dragons started to look too distorted, which makes sense as the model was trying to distort the input data to match the flare dataset. I compiled some of my favorites to display in the image at the start of this article, taken across several different epochs. I especially liked the dragons created around epoch 35, where the model’s distortions added fiery and wispy splashes of color to the dragon sketches. Below are a few more samples from epoch 35.

I was also impressed by how well the network learned to distort random colorful shapes as input images to form dragon-like drawings. Below are a few samples of dragon heads drawn by the network at the end of training.

Since 2021 is the year of the ox, I couldn’t end this article without a few festive oxen. I collected ~170 ox-like QuickDraw cow drawings and repeated the steps I described above to train CycleGAN on this new mini dataset. Training completed in about an hour and 40 minutes. Below are some of my favorite sample outputs.

Just for fun, I was curious about how the model trained on the dragon dataset would perform on an ox drawing input. Below is a sample using the epoch 35 dragon weights – a fire-breathing 🐲 + 🐂

Thank you for reading and, once again, GAN hay fat choy 🐉