A Gentle Introduction To Math Behind Neural Networks
Let’s dive into the Mathematics behind Neural Networks and Deep Learning
Today, with open source machine learning software libraries such as TensorFlow, Keras, or PyTorch we can create a neural network, even with high structural complexity, with just a few lines of code. Having said that, the mathematics behind neural networks is still a mystery to some of us, and having the mathematics knowledge behind neural networks and deep learning can help us understand what’s happening inside a neural network. It is also helpful in architecture selection, fine-tuning of deep learning models, hyperparameters tuning, and optimization.
Introduction
I had ignored understanding the mathematics behind neural networks and deep learning for a long time as I didn’t have good knowledge of algebra or differential calculus. A few days ago, I decided to start from scratch and derive the methodology and mathematics behind neural networks and deep learning, to know how and why they work. I also decided to write this article, which would be useful to people like me, who find it difficult to understand these concepts.
Perceptrons
Perceptrons — invented by Frank Rosenblatt in 1958, are the simplest neural network that consists of n number of inputs, only one neuron, and one output, where n is the number of features of our dataset. The process of passing the data through the neural network is known as forward propagation and the forward propagation carried out in a perceptron is explained in the following three steps.
Step 1: For each input, multiply the input value xᵢ with weights wᵢ and sum all the multiplied values. Weights — represent the strength of the connection between neurons and decides how much influence the given input will have on the neuron’s output. If the weight w₁ has a higher value than the weight w₂, then the input x₁ will have a higher influence on the output than w₂.
The row vectors of the inputs and weights are x = [x₁, x₂, … , xₙ] and w =[w₁, w₂, … , wₙ] respectively and their dot product is given by
Hence, the summation is equal to the dot product of the vectors x and w
Step 2: Add bias b to the summation of multiplied values and let’s call this z. Bias — also known as the offset is necessary in most of the cases, to move the entire activation function to the left or right to generate the required output values.
Step 3: Pass the value of z to a non-linear activation function. Activation functions — are used to introduce non-linearity into the output of the neurons, without which the neural network will just be a linear function. Moreover, they have a significant impact on the learning speed of the neural network. Perceptrons have binary step function as their activation function. However, we shall use sigmoid — also known as logistic function as our activation function.
where σ denotes the sigmoid activation function and the output we get after the forward prorogation is known as the predicted value ŷ.
Learning Algorithm
The learning algorithm consists of two parts — backpropagation and optimization.
Backpropagation: Backpropagation, short for backward propagation of errors, refers to the algorithm for computing the gradient of the loss function with respect to the weights. However, the term is often used to refer to the entire learning algorithm. The backpropagation carried out in a perceptron is explained in the following two steps.
Step 1: To know an estimation of how far are we from our desired solution a loss function is used. Generally, mean squared error is chosen as the loss function for regression problems and cross entropy for classification problems. Let’s take a regression problem and its loss function be mean squared error, which squares the difference between actual (yᵢ) and predicted value ( ŷᵢ ).
Loss function is calculated for the entire training dataset and their average is called the Cost function C.
Step 2: In order to find the best weights and bias for our Perceptron, we need to know how the cost function changes in relation to weights and bias. This is done with the help of the gradients (rate of change) — how one quantity changes in relation to another quantity. In our case, we need to find the gradient of the cost function with respect to the weights and bias.
Let’s calculate the gradient of cost function C with respect to the weight wᵢ using partial derivation. Since the cost function is not directly related to the weight wᵢ, let’s use the chain rule.
Now we need to find the following three gradients
Let’s start with the gradient of the cost function (C) with respect to the predicted value ( ŷ )
Let y = [y₁ , y₂ , … yₙ] and ŷ =[ ŷ₁ , ŷ₂ , … ŷₙ] be the row vectors of actual and predicted values. Hence the above equation is simplified as
Now let’s find the gradient of the predicted value with respect to the z. This will be a bit lengthy.
The gradient of z with respect to the weight wᵢ is
Therefore we get,
What about Bias? — Bias is theoretically considered to have an input of constant value 1. Hence,
Optimization: Optimization is the selection of the best element from some set of available alternatives, which in our case, is the selection of best weights and bias of the perceptron. Let’s choose gradient descent as our optimization algorithm, which changes the weights and bias, proportional to the negative of the gradient of the cost function with respect to the corresponding weight or bias. Learning rate (α) is a hyperparameter which is used to control how much the weights and bias are changed.
The weights and bias are updated as follows and the backpropagation and gradient descent is repeated until convergence.
Final Thoughts
I hope that you’ve found this article useful and understood the mathematics behind the neural networks and deep learning. I have explained the working of a single neuron in this article. However, these basic concepts are applicable to all kinds of neural networks with some modifications.
