How neural networks paint mountains and valleys — 3D hike
If you are like me, you would have come across hundreds of visualizations of neural network outputs and felt like something does not add up. Well, that’s because something is missing. A dimension, to be precise.
If you have come here directly, please check out the first story on the real output from a neuron. Once you have grasped one, it’s just about putting two and two together — both figuratively and literally!
OK, so let’s say we are tasked with a “saving the world” mission of finding a way to figure out if a virus if of the Coronavirus strand that causes Covid-19. While the politics and secrecy continues, you lay your hands on the ultimate database leaked on the dark web.
It is matrix of heat and radiation tolerance on the X and Y axis respectively. The pesky pandemic-causing virus population is in red while the other “relatively harmless” ones are in other harmless-looking green and yellow colors.
Let’s tackle this problem step at a time, or rather line at a time.
The red line gives us the first boundary, and we can can proudly report that the deadly Coronavirus can only be on the left side of the line (blue-shaded area). Anything on the right side (red-shaded area) is just a plain old who-cares virus.
We already know from our first story that classification using a line on a plane is nothing but the representation of a neuron with 2 inputs. It doesn’t solve the entire problem yet, but it definitely makes a good start!
The equation of this red line is “x-y-3=0”, but again from our first story in this series, we know better — a line is nothing but the intersection of 2 planes.
The output from the neuron is not a line, but in fact a plane that intersects the temperature-radiation plane on the red line. Here is what the neuron and its output without any activation looks like.
Now, let’s apply a simple sigmoid activation on top of this output and things start to get really interesting.
In the sigmoid non-linear output landscape:
Blue valley is where the Coronavirus may reside. Green slope is where we gradually climb from Corona-land to plain-old virus land. Red mountain is where only the plain-old virus stay.
If you look at the top view of this output, it would collapse back to 2D and look very similar to the chart with grid-lines we saw earlier. If there is only one thing you take away from this story, it should be the fact that even when you look at a neuron output in 2D, the colors actually represent the strength of the neuron’s output in 3D.
If you have followed everything to this point, we are ready to go deeper into the rabbit hole. Let’s add 3 new neurons to the problem and see what we come up with. So the bad Coronavirus is on the…
- left of the red line x-y-3=0
- right of the blue line -x+y-3=0
- left of the purple line x+y-3=0
- right of the green line x+y+3=0
Disclaimer: green-line is with no reference to any middle-eastern dispute :)
Just draw some circles and arrows using parameters from the colorful lines, and you transform the simple line-chart to a complex-looking neural network! We have already done this in our first story.
The output from 4 neurons put together is a delight to watch. The non-linear nature of the sigmoid function ensures that all the 4 individual outputs are able to bring their unique contribution to the final combined output.
The color-combination still holds but now defines the boundary much better. Blue valley is where the Coronavirus reside. Green slope is where we gradually climb from Corona land to plain-old virus land. Red mountain is where the plain-old virus stay.
And finally, the view from the top of this wonderful landscape.
As mentioned earlier, the color represents the height of the landscape, which in turn represents the strength of the output from the neural network.
An unimaginative poem would read “Red is high, blue is low, green where the climb is slow”
Here is the side-by-side comparison of the 2D view and the 3D top-view. This ANN output is by no means the most optimal, but it should help you develop an intuition about what is actually happening under the hood anytime you look at a neural network classification output.
As you can imagine, the “top view” of the output from a neural network with 3 input dimensions would be represented by a sphere blue at the center and then turning to green to red as you move outwards.
I will leave you with the final image which uses a step activation instead of a sigmoid. Sure enough, the landscape disappears and we are left with only cliffs! Not surprising, the 3D top-view is no different from the 2D view as the step function jumps straight from the floor to the ceiling.
Bonus reading: If you want to learn more about the magic non-linearity brings to neural networks to create these amazing shapes, this 4-min read in this ongoing series would be a good start.
