Denoising Autoencoders explained
Last month, I wrote about Variational Autoencoders and some of their use-cases. This time, I’ll have a look at another type of Autoencoder: The Denoising Autoencoder, which is able to reconstruct corrupted data.
Autoencoders are Neural Networks which are commonly used for feature selection and extraction. However, when there are more nodes in the hidden layer than there are inputs, the Network is risking to learn the so-called “Identity Function”, also called “Null Function”, meaning that the output equals the input, marking the Autoencoder useless.
Denoising Autoencoders solve this problem by corrupting the data on purpose by randomly turning some of the input values to zero. In general, the percentage of input nodes which are being set to zero is about 50%. Other sources suggest a lower count, such as 30%. It depends on the amount of data and input nodes you have.
When calculating the Loss function, it is important to compare the output values with the original input, not with the corrupted input. That way, the risk of learning the identity function instead of extracting features is eliminated.
A great implementation has been posted by opendeep.org where they use Theano to build a very basic Denoising Autoencoder and train it on the MNIST dataset. The OpenDeep articles are very basics and are made for beginners. So even if you don’t have too much experience with Neural Networks, the article is definitely worth checking out!
Denoising Autoencoders are an important and crucial tool for feature selection and extraction and now you know what it is! Enjoy and thanks for reading!
