— Augmenting human creativity with artificial intelligence

Mixing, creating and searching aesthetically coherent designs with neural networks.

One of the most discussed aspects of artificial intelligence and computer science its whether machines can be creative or not. This discussion is as old as the first computer, but in recent times amazing results from Generative Adversarial Networks and similar architectures really made the discussion bigger. 
Being a creative and a technologist, I recently focused on a similar, but intrinsically different topic: can AI augment human creativity? Can a designer enhance its inspiration using machine intelligence?

In the beginning, there was Paint. One of the firsts, if not the first, software instruments to help creatives be creative. Then, people started to see how computers were really cool to do graphic stuff. Now, we have Adobe softwares, such as Photoshop, that tremendously enhance a human ability to create artworks and edit pictures, in ways that were unthinkable before. What’s the next step? Artificial Intelligence. To analyze this new field, I focused on two different areas of design: interior design and fashion design.

Inspiration, as it’s known, it’s a muse. It’s unpredictable. Great ideas can come out of nowhere, and can be nowhere to be found even if one focuses for weeks. So I worked on different aspects that can affect inspiration: research and creation.

  • Research means finding designs from the past, searching for something similar to what we have in mind to get inspiration.
  • Creation means going from zero to a finished design in as little time as possible, thus creating tools that allow fast creative experiments.

So, the goal was simple: to revolutionize the way designers work with artificial intelligence.

As a first step, I created a small dataset of chairs and sofas from the internet (around 500 samples). I’ve always loved simple interior design, and the idea of merging AI with interior design really interested me. I needed to implement a neural network architecture that could extract visual features from this unlabeled data, and that was basically able to understand how a chair looks like. Thus, I implemented a convolutional auto encoder, which learns how to compress the original image into a smaller size (in my case, from 240x240 to 15x15), and then how to reconstruct the original image as closely as possible.

Convolutional auto encoder structure.

The bottleneck structure forces the network to learn important visual features that are numerically represented in the latent space of dimension 15x15. This was the core model behind the applications that I developed, and that I will discuss later in this post.

Reconstruction of chair designs with convolutional autoencoders.

What is the advantage of creating and training an auto encoder structure? The core is in the latent space representation. This small matrix contains, in an often unpredictable way, salient visual features of the dataset that the network has learned to recognize, and it’s computationally better to work in this space to compute, for examples, distances, or even making algebraic summations.

How can these tools allow a designer to easily and intuitively search in a dataset of chair designs or textile/clothing design? Projecting all the dataset in this latent space allows us to easily compute distances between samples and thus doing reverse image search. While there are many ways to accomplish this, I was curious to check the performance of a “neural” reverse image search, in my case using chairs. Below some results are shown: you can choose an image as query; the network (in particular, the encoder) extracts the latent representation, and then you can compute the distance (euclidean, cosine, etc.) between the query and the dataset images. This makes the process of finding inspiration having a reference design profoundly easier.

Neural Inverse Image Search. The first chair (top) is the query, will the other two are two random samples that are similar enough. (latent space distance under a threshold)

Another interesting application is mixing different samples from the dataset to generate new ones that share visual features from both. In particular, it’s possible to mix the design of different chairs to obtain a new one that is aesthetically coherent and shares design details with both chairs. This, again, allows a designer to immediately experiment new ideas by mixing interesting design, until some result sparkles his/her creativity. This is possible by averaging the latent space representation of different chair design. A weighted average can even give access to a great number of nuances, i.e. creating a new design that is closer to one or another “parent”. Check out some interesting results below.

Mixing different chair designs to generate a new one.

I also trained a Deep Convolutional Generative Adversarial Network to create new chair designs from scratch, but the small dataset didn’t allow to obtain really good results, while generally overfitting some dataset samples. Furthermore, this method doesn’t directly allow to mix up different existing designs.

A DCGAN dreaming of chairs.

Inspired by the pix2pix approach, I trained the same architecture to take drawings of chairs as input , which are basically the automatically computed edges of the images in the dataset, and to reconstruct from them the original pictures. Doing so, the network can be trained to generate designs starting from simple drawings, made on a tablet or even with the mouse. Since my dataset was quite small (the datasets usually adopted have ~10k samples, while mine has around 500), the results aren’t outstanding, but they are promising: from a simple sketch made on Gimp with a trackpad, the network can generate something that resembles a chair design. It’s interesting how it automatically modifies, corrects and fills some part of the drawing, based on its experience of how a chair should look like.

An example of generating a chair from a drawing. Notice how the network modifies and corrects some parts of the sketch.

I applied the same techniques on fashion design, in particular textile design, with similar results. I developed further this branch of the project with a couple of friends of mine, and we created a neural reverse image search that works with simple drawings, made on a web app. In the demo, a user can simply sketch a pattern that he/she would like to find in a database of textile patterns. The neural network extracts visual features from the drawing and compares them with ones from the database images. After some computation, it returns the ones who seem to match the most.

Neural reverse image search directly from drawings.

This is a really simple and intuitive way of searching in a database of patterns, and can save any designer tons of time.

The textile patterns database that we extracted from the web is also a fun playground to test the mixing of designs to create a new one. In a couple of seconds, designers can create new patterns by combining existing ones and gather a lot of ideas and inspiration.

Neural pattern mixing. In a couple of seconds, a neural network creates new pattern starting from existing ones.

Combining AI and design is truly inspiring for me, being very passionate about both. I truly believe that AI can create the augmented artist, enabling new ways of searching and creating. People of the field seem to like this idea, because thanks to this project we won CognitiveHack, an hackathon recently held in Milan. The challenge was about fashion design, and ways of augmenting designers ability and inspiration. This idea was awarded as the best one, and will be furtherly expanded in the future in collaboration with fashion brands and companies who sponsored the event.