Hands-on Tutorials

Animations of Gradient Descent and Loss Landscapes of Neural Networks in Python

Tobias Roeschl
Towards Data Science
9 min readDec 21, 2020

During my journey of learning various machine learning algorithms, I came across loss landscapes of neural networks with their mountainous terrains, ridges and valleys. These loss landscapes were looking very different from the convex and smooth loss landscapes I encountered with linear and logistic regression. In the following article, we will create loss landscapes of neural networks and animate gradient descent using the MNIST-dataset.

Fig. 1: Loss landscape of a convolutional neural network with 56 layers (VGG-56, source¹)

The above image exemplarily depicts the highly non-convex loss landscape of a neural network. A loss landscape is a visual representation of values a cost function takes on for a given range of parameter values given our training data. Since our objective is to visualize the costs in three dimensions, we need to choose two particular parameters to be varied for our plots, while all other model parameters are kept fixed. It is worth mentioning, however, that there are more advanced techniques (e.g. dimensionality reduction, filter normalization) that can be used to approximate neural network loss landscapes in a low-dimensional parameter subspace.¹ A three-dimensional representation of the loss landscape of a VGG neural network with 56 layers is displayed in figure 1. However, this is beyond the scope of this article.

Create an account to read the full story.

The author made this story available to Medium members only.
If you’re new to Medium, create a new account to read this story on us.

Or, continue in mobile web

Already have an account? Sign in

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Written by Tobias Roeschl

Resident physician passionate about data science, statistics and machine learning. Get in touch: www.linkedin.com/in/tobi-roeschl-60430217b

Responses (5)

What are your thoughts?

Recommended from Medium

Lists

See more recommendations