What is the mathematical foundation of generative adversarial networks?
Generative Adversarial Networks or GANs have been a revolution in Deep Learning over the last decade. Since the first version of GAN that was released in 2014 by Ian Goodfellow et al., there have been numerous variants of GAN with some mind-blowing applications.
In this article we shall try to understand the basic mathematical foundation behind a GAN in simple terms.
To get a high-level intuition of how Gans work, it is recommended that you first browse through the following article: An intuitive introduction to GANs
GAN Overview

The generator takes a random vector [z] as input and generates an output image [G(z)]. The discriminator takes either the generated image [G(z)] or a real image [x] as input and generates an output[D].
During the discriminator training, the generator’s weights and biases are frozen. The discriminator is penalised when it predicts a real image as fake or a fake image as real. The weights and biases of the discriminator are updated through back propagation from the discriminator loss.
Similarly during the generator training, the discriminator’s weights and biases are frozen. The goal of the generator is to deceive the discriminator. Hence the generator is penalised if the discriminator can identify that the image generated by the generator is a fake image. In this case the loss is back propagated through the generator network to update its weights and biases.
The two-player MiniMax Game
A GAN can be conceptually thought of as a minimax game played between the generator model and the discriminator model. Both models are trained simultaneously where one model tries to minimise the loss while the other tries to maximise the loss.
As per the original GAN paper, the loss function for GAN is as below

Let us try to derive this equation to understand it better.
Binary Cross-Entropy Loss
Both the generator and the discriminator use the binary cross-entropy loss to train the models. In general, binary cross entropy loss can be written as:

Discriminator Loss
The discriminator can have two possible inputs, real or fake.
- For a real input, y = 1. Substituting this in the binary cross entropy loss function gives the below loss value:

- Similarly for a fake input, y = 0, which gives the below loss value:

Combining both the above losses for the discriminator, one for a real input and the other for a fake input gives the below discriminator loss

The goal of the discriminator is to minimise this loss,

Removing the negative sign will change min to max, hence the final discriminator loss for a single datapoint can be written as:

Generator Loss
For the generator, loss is calculated from the discriminator loss. During the training of the generator, the discriminator is frozen. Hence only one input is possible to the discriminator, which is the fake input. This nullifies the first term in the discriminator loss equation to 0.
The generator is trying to fool the discriminator into classifying the fake data as real data. This implies that the generator tries to minimise the second term in the discriminator loss equation. The generator loss function for single generated datapoint can be written as:

GAN – Loss Equation
Combining both the losses, the discriminator loss and the generator loss, gives us an equation as below for a single datapoint.

This is the minimax game played between the generator and the discriminator. The generator tries to minimise the loss whereas the discriminator tries to maximise the loss.
Generalising this equation to the expected values (E) for multiple datapoints gives the final equation as below,

Now compare this with the loss function given in the GAN paper to understand it better.

Limitations of GAN
Here are some of the commonly faced issues in GAN
- Vanishing gradient issue due to which the generator training might fail
- Model collapse where the generator might repeatedly create the same output
- Failure to converge due to which the discriminator feedback can get less meaningful to the generator thus impacting its quality
Conclusion
There has been lot of improvements in GANs since its inception and researchers have identified ways to overcome these limitations to a large extent. Choosing the right loss function is one of the hot research topics and many alternatives are being proposed and evaluated.
Hope you enjoyed this article. Have fun in diving deeper into the various flavours of GAN.