Introduction To Autoencoders
A Brief Overview
Autoencoders are neural network-based models that are used for unsupervised learning purposes to discover underlying correlations among data and represent data in a smaller dimension. The autoencoders frame unsupervised learning problems as supervised learning problems to train a neural network model. The input only is passed a the output. The input is squeezed down o a lower encoded representation using an encoder network, then a decoder network decodes the encoding to recreate back the input.
The encoding produced by the encoder layer has a lower-dimensional representation of the data and shows several interesting complex relationships among data.
An Autoencoder has the following parts:
- Encoder: The encoder is the part of the network which takes in the input and produces a lower Dimensional encoding
- Bottleneck: It is the lower dimensional hidden layer where the encoding is produced. The bottleneck layer has a lower number of nodes and the number of nodes in the bottleneck layer also gives the dimension of the encoding of the input.
- Decoder: The decoder takes in the encoding and recreates back the input.
The bottleneck layer is the lower dimension layer. In the diagram, we have the neural networks encoder and decoder. Phi and Theta are the representing parameters of the encoder and decoder respectively.
The target of this model is such that the Input is equivalent to the Reconstructed Output. To achieve this we minimize a loss function named Reconstruction Loss. Basically, Reconstruction Loss is given by the error between the input and the reconstructed output. It is usually given by the Mean Square error or Binary Crossentropy between the input and reconstructed output. Binary Crossentropy is used if the data is binary.
Now, we have a basic understanding of encoders. So, let’s understand a basic tradeoff we need to know while designing an autoencoder. We have to keep in mind that the reason behind using an autoencoder is that we want to understand and represent only the deep correlations and relationships among data. We need a generalized lower-dimensional representation. That is why, if the features of the data are not correlated at all then it is hard for an autoencoder to represent the data in a lower dimension. If while designing the neural network, we use a very large number of nodes in the bottleneck layer, it will create a large dimensional encoding. The problem that exists here is, the network might cheat and overfit to the input data by simply remembering the input data. In this case, we will not be able to get the correct relationships in our encodings. Again, if we use a shallow network with a very less number of nodes, it will be very hard to capture all the relationships. So, we must be very careful during designing the network.
Now, a question may arise, why go for autoencoder, when we have methods like PCA for dimensionality reduction?
Well, here goes the explanation. PCA or principal component analysis tries to find lower-dimensional orthogonal hyperplanes that describe the original data by capturing the maximum possible variance in the data and the important correlations consequently. We need to focus on the fact that we are talking about finding a hyperplane, so it's linear. But often correlations are non-linear, which are not covered by PCA.
As we can see in the above diagram autoencoders cover non-linear data dependencies, thus are a better way than PCA for dimensionality reduction.
Let’s look at some of the applications of autoencoders:
- Autoencoders are used largely for anomaly detection: As we know, autoencoders create encodings that basically captures the relationship among data. Now, if we train our autoencoder on a particular dataset, the encoder and decoder parameters will be trained to represent the relationships on the datasets the best way. Thus, will be able to recreate any data given from that kind of dataset in the best way. So, if data from that particular dataset is sent through the autoencoder, the reconstruction error is less. But if some other kind of data is sent through the autoencoder it will generate a huge reconstruction error. If we are able to apply a correct cutoff we will be able to create an anomaly detector.
- Autoencoders are used for Noise Removal: If we can pass the noisy data as input and clean data as output and train an autoencoder on such given data pairs, trained Autoencoders can be highly useful for noise removal. This is because noise points usually do not have any correlations. Now, as the autoencoders need to represent the data in the lowest dimensions, the encodings usually have only the important relations there exists, rejecting the random ones. So, the decoded data coming out as output of an autoencoder is free of all the extra relations and hence the noise.
- Autoencoders as Generative models: Before the GAN’s came into existence, Autoencoders were used as generative models. One of the modified approaches of autoencoders, variational autoencoders are used for generative purposes.
- Autoencoders used for collaborative filtering: Collaborative filtering normally uses matrix factorization methods, but autoencoders can learn the dependencies and learn to predict the item-user matrix
Types of Autoencoders
Several kinds of Autoencoders have been developed to answer the different tradeoffs. Let’s look at some of them.
Undercomplete Autoencoder
The undercomplete autoencoders are the simplest architecture for autoencoders. The architecture depends on putting constraints on the number of nodes that can be added to the hidden layers and the central bottleneck. The theory behind this is, the approach tries to restrict the flow of information through the network. The architecture depends on the fact, that if the flow of information is less and the network needs to learn the encoding the best way, it will only consider the most important dependencies and reject the rest. Thus we will be able to create the encoding for best reconstruction.
The loss function used is normal reconstruction error loss, which is MSE or Binary Crossentropy. As we are restricting the flow of information using the bottleneck, there is no chance that the model memorizes the input and cheats.
The above diagram shows an undercomplete autoencoder. We can see the hidden layers have a lower number of nodes.
Let’s see the application of TensorFlow for creating undercomplete autoencoder.
Simple Undercomplete autoencoder:
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense,Input(x_train,_), (x_test,_)= mnist.load_data()x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255x_train=x_train.reshape((len(x_train),np.prod(x_train.shape[1:])))
x_test=x_test.reshape((len(x_test),np.prod(x_test.shape[1:])))## Shape of the x_train is (60000, 784)
## Shape of the x_test is (10000,784)input_l=Input(shape=(784,))
bottleneck=Dense(32, activation='relu')(input_l)
output_l=Dense(784, activation='sigmoid')(bottleneck)autoencoder=Model(inputs=[input_l],outputs=[output_l]) ## Building the entire autoencoderencoder=Model(inputs=[input_l],outputs=[bottleneck]) ## Building the encoderencoded_input=Input(shape=(32,))
decoded=autoencoder.layers[-1](encoded_input)decoder=Model(inputs=[encoded_input],outputs=[decoded]) ##Building the decoderautoencoder.compile(optimizer='adam', loss='binary_crossentropy')autoencoder.fit(x_train, x_train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
The models created by the above code are:
The first model is the decoder, the second is the full autoencoder and the third is the encoder model. The bottleneck layer is the place where the encoded image is generated.
We use the autoencoder to train the model and get the weights that can be used by the encoder and the decoder models.
If we send image encodings through the decoders, we will see that the images are reconstructed back.
The upper row is the original images and the lower row is the images created from the encodings by the decoder.
Now, the images are of dimensions 28x28, and we have created encodings of dimensions of 32. if we represent the encodings as 16x2, it will look something like this:
The lower row represents the corresponding encodings.
As we can see here, we have built a very shallow network, we can build a deep network, as the shallow networks may not be able to uncover all the underlying features, but we need to be very careful about restricting the number of hidden nodes.
input_l=Input(shape=(784,))encoding_1=Dense(256, activation='relu')(input_l)
encoding_2=Dense(128, activation='relu')(encoding_1)bottleneck=Dense(32, activation='relu')(encoding_2)decoding_1=Dense(128, activation='relu')(bottleneck)
decoding_2=Dense(256, activation='relu')(decoding_1)output_l=Dense(784, activation='sigmoid')(decoding_2)autoencoder=Model(inputs=[input_l],outputs=[output_l])encoder=Model(inputs=[input_l],outputs=[bottleneck])encoded_input=Input(shape=(32,))decoded_layer_1=autoencoder.layers[-3](encoded_input)
decoded_layer_2=autoencoder.layers[-2](decoded_layer_1)decoded=autoencoder.layers[-1](decoded_layer_2)decoder=Model(inputs=[encoded_input],outputs=[decoded])
The above implemented as a deep undercomplete network.
Sparse Autoencoders
When we were talking about the undercomplete autoencoders, we told we restrict the number of nodes in the hidden layer to restrict the data flow. But often this approach creates issues because the limitation on the nodes of the hidden layers and shallower networks prevent the neural network to uncover complex relationships among the data items. So, we need to use deeper networks with more hidden layer nodes. Again, if we use more hidden layer nodes, the network may just memorize the input and overfit, which will make our intentions void. So, to solve this we use regularizers. The regularizers prevent the network from overfitting to the input data and prevent the memorization problem.
During regularization, we normally regularize weights but in this case, we regularize activations that are actually passed from one hidden layer to another. In simpler words, the idea is we won’t let all the nodes in the hidden layers learn. Now, if we go to the basics of neural networks, an activation function controls how much information a particular node passes. The activation function works like a gate. If the activation for a particular node is 0, then the node is not contributing its information. The idea of sparse autoencoders is something like that.
Now, one thing to note is, the activations are dependent on the input data ad will change with the change in input. So, we let our model decide the activations and penalize their activation values. We usually do this in two ways:
L1 Regularization: L1 regularizers restrict the activations as discussed above. It forces the network to use only the nodes of the hidden layers that handle a high amount of information and block the rest.
It is given by:
The reconstruction loss is given by L and the second part is the regularizers that penalize the activations. As we can see the regularizer part is a summation of activations of all nodes in the hidden layer h. So, when we try to minimize the loss function we decrease the activations. Again, we use a tuning parameter lambda. Lambda helps to ensure how much attention we want to pay for the regularization aspect.
KL Divergence: Kullback-Leibler Divergence is a way to measure the difference and similarity between two mathematical probability distributions. It is given by:
So, basically, it tells us how similar p and q are. This method uses a sparsity parameter ρ (Rho). Rho is said to be the average activation of a neuron over a set of samples. The idea is to use a very low Rho value such that the neuron or the nodes keep a low value as average and in order to achieve that the node will have just 0 activations for some of the samples in the collection, where it is not essential.
Now, the question is how does the KL divergence help. For this, we will need to know what is a Bernoulli Distribution.
In probability theory and statistics, the Bernoulli distribution, is the discrete probability distribution of a random variable which takes the value 1 with probability p and the value 0 with probability q=1-p
So, basically it a binary level probability distribution. We want something similar to our nodes. We want it to fire with a probability and so its distribution can be similar to a Bernoulli distribution. Now, for a particular neuron j we can calculate Rho as:
where m is the number of observations and a is the activation of the neuron in the hidden layer h. The loss is given by:
A visualization will look like this:
The above image shows the light red nodes do not fire.
Let’s see the application of TensorFlow for creating a sparse autoencoder.
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense,Input
from tensorflow.keras.regularizers import l1(x_train,_), (x_test,_)= mnist.load_data()x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255x_train=x_train.reshape((len(x_train),np.prod(x_train.shape[1:])))
x_test=x_test.reshape((len(x_test),np.prod(x_test.shape[1:])))## Shape of the x_train is (60000, 784)
## Shape of the x_test is (10000,784)input_l=Input(shape=(784,))encoding_1=Dense(256, activation='relu', activity_regularizer=l1(0.001))(input_l)
bottleneck=Dense(32, activation='relu', activity_regularizer=l1(0.001))(encoding_1)
decoding_1=Dense(256, activation='relu', activity_regularizer=l1(0.001))(bottleneck)
output_l=Dense(784, activation='sigmoid')(decoding_1)autoencoder=Model(inputs=[input_l],outputs=[output_l])encoder=Model(inputs=[input_l],outputs=[bottleneck])encoded_input=Input(shape=(32,))
decoded_layer_2=autoencoder.layers[-2](encoded_input)
decoded=autoencoder.layers[-1](decoded_layer_2)decoder=Model(inputs=[encoded_input],outputs=[decoded])autoencoder.compile(optimizer='adam', loss='binary_crossentropy')autoencoder.fit(x_train, x_train,
epochs=50,
batch_size=256,
shuffle=True,
validation_data=(x_test, x_test))
The above code uses an L1 regularizer.
The images represent the full autoencoder, followed by the encoder and the decoder.
Denoising Autoencoders
We already have talked about autoencoders used as noise removers. So, the idea is in order to represent the underlying relations and represent in the small size encoding the autoencoders only look at the object image and not the noise, which is eliminated. Again, here we do not need to restrict the number of nodes or use a regularizer as we have a different input and output and the memorization problems do not exist anymore.
The above network represents denoising autoencoders.
input_l=Input(shape=(28,28,1))encoding_1=Conv2D(32, (3,3), activation='relu',padding='same')(input_l)
maxp_1=MaxPooling2D((2,2), padding='same')(encoding_1)encoding_2=Conv2D(16, (3,3), activation='relu',padding='same')(maxp_1)
maxp_2=MaxPooling2D((2,2), padding='same')(encoding_2)encoding_3=Conv2D(8, (3,3), activation='relu',padding='same')(maxp_2)bottleneck=MaxPooling2D((2,2), padding='same')(encoding_3)decoding_1=Conv2D(8, (3,3), activation='relu', padding='same')(bottleneck)
Up_1=UpSampling2D((2,2))(decoding_1)decoding_2=Conv2D(16, (3,3), activation='relu', padding='same')(Up_1)
Up_2=UpSampling2D((2,2))(decoding_2)decoding_3=Conv2D(32, (3,3), activation='relu')(Up_2)
Up_3=UpSampling2D((2,2))(decoding_3)output_l= Conv2D(1,(3,3),activation='sigmoid',padding='same')(Up_3)autoencoder=Model(inputs=[input_l],outputs=[output_l])encoder=Model(inputs=[input_l],outputs=[bottleneck])autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
The above code can be used to create the autoencoder.
The results on datasets are as follows:
The above are the results of the fashion mnist datasets.
As you can see, I have used a convolutional network to create the autoencoder.
The model structure is as below:
Contractive Autoencoders
The principle that the contractive autoencoders are based on is pretty similar to the denoising encoders. The idea is that the encodings produced for similar inputs will be similar. In other words, if we change the inputs or tweak them by just a little the encodings will remain the same and show no changes. They are used for feature extractions.
Autoencoders can be implemented using any kind of neural network, like for image data we can use Convolutional Neural Nets and for time series data we can use Recurrent Neural Nets.
There exists another type of autoencoders that are a bit different from the above-stated ones which are called Variational Autoencoders.
Variational Autoencoders
To understand the concept we need to dive into a bit of basic. Actually, what do we mean by the lower dimensional encoding? The answer is, as we have seen above our above input had 784x1 or 28x28 dimension, when we encode it to a say much smaller 32x1 dimension, we basically mean that now we have 32 features which are the most important features and reflect most of the information in the data, or image.
So, say for a face, when we encode a face image of say 32x32 dimension, it has the full facial two-dimensional image, now, if we encode it to 6x1 dimension, i.e, send it through a bottleneck layer of 6 nodes, we will basically get 6 features which contribute most or the major information about the facial image. Say, for the 6 features we have a smile, skin tone, gender, beard, wears glasses, and hair color. Our encoding has a numerical value for each of these features for a particular facial image. By sending the encodings through a decoder we can reconstruct back the image.
The 6 features we talked about in the lower dimension encoding are called latent features/attributes and the set of values feature can take is its latent space.
Now, different values of the latent attributes represent different images as the feature varies as shown below.
We have seen that the values of the latent attributes are always discrete. This is where the variational autoencoders are different. Instead of considering to pass discrete values, the variational autoencoders pass each latent attribute as a probability distribution. Some thing as shown below.
So, our representing face example becomes as follows:
Now, we can see each latent attribute is passed as a probability distribution. The decoder samples from each latent distribution and decodes to reconstruct the image. So, as the sampling is random and not backpropagated the reconstructed image is similar to the input but is not actually present in the input set. This is the reason for variational autoencoders to be known as a generative network.
The above image defines the situation. Now, to create a distribution for each latent vector, the encoder in place of passing the value, pass the mean and standard deviation of the distribution, which is used to create construct the normal distribution.
The above image shows the structure of a variational autoencoder. The Probabilistic encoder is called the recognition model and the decoder is called the generative model.
Now, as the z or the latent values are sampled randomly, they are unknown and hence called hidden variables. Again, we know the goal is such that our reconstructed output is equivalent to the input. So, our goal is to find out what is the probability of a value to be in z or the latent vector given that it is similar to x, P(z|x), because actually we need to reconstruct x from z. In simpler words, we can see x but we need to estimate z.
We obtain the above equation, using bayes theorem. This method requires finding p(x), given by:
This problem is untractable or it won’t complete in polynomial time as, it is a multiple integral problem and the number of integral increases with the increase in latent attributes or encoding dimensions.
In order to solve this problem, we use another distribution q(z|x) which is the approximation of p(z|x) and is designed to be a tractable solution. Now, again to determine the fact that q(z|x) is similar to p(z|x) we use KL divergence between the two distributions.
The variational autoencoders use a loss function as:
The first term is the reconstruction error and the second term is the KL divergence between the two distributions. It ensures that distributions are similar, as it minimizes the KL divergence to minimize the loss. We use a trick called the Reparameterization Trick to resample latent points from distributions without using back propagation.
Conclusion
We have seen all types of autoencoders there exist and their uses.
The codes are available here.
I hope this helps.
