The world’s leading publication for data science, AI, and ML professionals.

Drawing Anime Girls With Deep Learning

This article is about a side project by Mary Kate MacPherson. We like to do side projects like AI for music videos, and the party button…

This article is about a side project by Mary Kate MacPherson. We like to do side projects like AI for music videos, and the party button. The idea was to try out an adversarial neural network that generates new anime faces after training on a set of human-created anime faces. We started from an online example. The original code for that work and the anime faces dataset can be found here.

Running training on the original code from here.
Running training on the original code from here.

As you can see in the image above, the Generative Adversarial Network (GAN) iteratively improves over time at generating realistic images. A GAN contains a generator that makes new images, and a discriminator that gives the generator constructive feedback. Both the generator and discriminator are Convolutional Neural Networks (CNNs).

Here is an example of the training data:

The following image is a nice example of the final output.

Anime faces generated entirely using artificial intelligence.
Anime faces generated entirely using artificial intelligence.

In the image above you can see that the bottom right character has a bow on her head, and the eyes and mouth are usually in the correct location on the face. Some generated outputs are kind of mushed as we see in the first row second from the left. The images look good in general, although they are not very big (64 by 64 pixels for 3 color channels).

This was a cool result, but there are some known limitations to CNNs. For example, Capsnets are better at understanding the placement of things in a picture than CNNs (see image below).

CNNs do not mind "seeing" faces when everything is in the picture, but in weird places (source [here](https://medium.com/ai³-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b) and further reading here). This is a problem capsule networks aim to address.
CNNs do not mind "seeing" faces when everything is in the picture, but in weird places (source [here](https://medium.com/ai³-theory-practice-business/understanding-hintons-capsule-networks-part-i-intuition-b4b559d1159b) and further reading here). This is a problem capsule networks aim to address.

Now, being an overachiever, Mary Kate MacPherson decided to try this task using a GAN with a capsule network as the discriminator instead of a CNN. The dataset was moved over from the original code base to the capsule network code base. The capsule network code came from here, and the generator from the capsule network code was replaced by the generator from the anime project, so as to make anime girls rather than handwritten numbers.

Running training on the new neural architecture with capsule network.
Running training on the new neural architecture with capsule network.
Final output of the capsule network GAN. Some are really good. Others, like the second from left on the first and last rows, don't have the right face parts.
Final output of the capsule network GAN. Some are really good. Others, like the second from left on the first and last rows, don’t have the right face parts.

We didn’t see a big improvement using a capsule network as the discriminator in place of the CNN. Perhaps generating and discriminating statistical features of the image will give us better results (idea from nVidia research paper here). We are working on it. We added a dimension to the generator’s prediction that generates the Histogram of Oriented Gradients (HOG) of the generated image. HOG is used to help detect objects, and so it seemed like a good fit for something we hope will teach the generator neural network to make better predictions of real faces. We are also looking at cleaning up the dataset. Mary Kate MacPherson will post the results of that adventure to a dedicated twitter channel on twitter when we have some good results. If you have your own generator going, we are happy to post your results as well. So far the new approach is not looking super promising, but that’s how these side projects go: It doesn’t work until suddenly it does work.

Future work idea 2: GANs with multiple capsule networks!

And so, in conclusion, Artificial Intelligence can be used to generate art and to create new images in the style of a set of existing images.

If you liked this article on generating anime girl faces, press the follow button and have a look at some of my most read past articles, like "How to Price an AI Project" and "How to Hire an AI Consultant." In addition to business-related articles, I also have prepared articles on other issues faced by companies looking to adopt deep machine learning, like "Machine learning without cloud or APIs."

Happy Coding!

-Daniel [email protected] ← Say hi. Lemay.ai 1(855)LEMAY-AI

Other articles you may enjoy:


Related Articles