Looking back a decade (2010–2020), a four part series
Rise of Neural Networks
Neural Networks have been outperforming many other machine learning models like SVM, adaboost, gradient boosted trees etc., in a variety of tasks like image recognition, visual question answering, sentence completion etc., establishing itself as the the new state of the art in many machine learning problems. Multiple researches over the years have proved the effectiveness of neural networks in learning implicit patterns in data. But it is largely unknown or sometimes even mysterious as to what exactly a neural network learns and how it makes its decisions.
Explainability
Images are fairly easy for humans to understand. For instance, humans can easily point out the object of interest within the image, distinguish background from foreground and list distinct objects of interest within the image.
On the other hand, machine learning models look at images as only a rectangular array of pixel values, with no inherent representation of shapes and hues. Despite this limitation, what makes neural networks so good at recognizing images when they are trained on a large collection of labelled images?
What is this blog about?
Parallel to the developments in finding better performing, easier to train deep neural networks in the decade 2010–2020, more and more researchers have also been interested in developing insights into what a neural network learns and how it makes it decisions.
This blog focusses on this later track of work, highlighting important advancements in understanding and explaining the decisions made by a neural network, especially regarding image understanding. We describe the developments in this domain in a roughly chronological order, showing the flow of ideas across the years.
We divide our presentation into a four part blog series:
- Part 1 talks about the effectiveness of Visualizing Gradients of the image pixels for explaining the pre-softmax class score of CNNs.
- Part 2 talks about some more advanced/modified gradient based methods like DeConvolution, Guided Back Propagation for explaining CNNs.
- Part 3 talks about some short comings of gradient based approaches and discusses alternate axiomatic approaches like Layer-wise Relevance Propagation, Taylor Decomposition, Deep LiFT.
- Part 4 talks about some recent developments like Integrated Gradients (continuing from part 3) ** and recent novelties in CNN architecture like Class Activation Map**s developed to make the the feature maps more interpretable.
Gradient Based Approaches
Given a model’s scoring function for a class, which of the inputs has the most influence over the score? One natural answer to this question is the gradient of the score with respect to each input. Higher the gradient, more sensitive the score is to the change in the respective input pixel.
Visualizing Gradients (2014)
In general, can we use gradients of the class activation score (pre-softmax) with respect to the individual pixels of the image to gain some insight on which pixels play the most important role in the decision making?
This question was first studied by Karen Simonyan, Andrea Vedaldi and Andrew Zisserman in their work "Deep Inside Convolutional Networks: Visualizing Image Classification Models and Saliency Maps, (ICLR 2014)". Since then, most works on explainable neural networks has employed some version of using gradients or defining gradient-like candidates which can be back propagated down to the inputs for understanding the decisions of the network.
The authors address two important questions in their work:
1. Saliency Map: Given an image, which pixels within the image are the most important in helping the network make the decision?
For the first question, the authors propose, visualizing the gradients of the class activation score (pre-softmax) with respect to the image pixel:



2. Class Model: How does a typical "dog" or "cat" or any such target image class look like according to the neural network? For the second question, the authors propose, finding the optimal image, constrained on its brightness, that would maximize the class score:


SmoothGRAD (2017)
Going back to the ideas of Simonyan et al on visualizing gradients, it was often observed that gradient visualizations were often noisy. Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viegas and Martin Wattenberg suggest some simple tricks to make gradient visualization better in their work "SmoothGrad: removing noise by adding noise (ICML 2017)". Here are some of their suggestions:
- Add a small gaussian noise to the input image and sample multiple images from this distribution. After calculating gradients, average the gradients out. This cleans out the noise in the gradient visualization, highlighting only the important pixels which would have been activate across all the sampled images.
- Image recognition tasks are invariant under color and illumination changes. For example, consider a black ball in white background vs a white ball in black background. In such cases, visualizing absolute value of gradients is often enough. One notable exception is the MNIST digit recognition, where all images are black against white background. In such cases, the sign of gradients becomes more important.
- It has been found across multiple researches that gradient maps often have a few pixels with very high gradient values. Capping them off improves visualizations.
- Sometimes, multiplying the gradients with the image provides sharper visualizations. But this can sometimes have undesirable effects of making darker pixels look less important than what the original gradients would show.
- All the above improvisations can also be used on any gradient based methods like Guided BackPropagation (described in Part 2).
Here are some results presented by the authors:


The authors note that SmoothGRAD outperforms guided back propagation when the image is against a constant background, while guided back back propagation performs better when the image has a textured background. Also, the smoothing methods when used along with with guided back propagation enhance the resulting visualizations.
Whats Next?
Continuing from gradient based methods, researchers developed more advanced methods like DeConvolution and Guided Back propagation. These methods improve upon "vanilla" gradient back propagation to provide cleaner visualizations. We discuss these cool techniques in the next part.
To read more about such exciting works on Explainability of neural networks, you can catch the next part here: Link to Part 2