The world’s leading publication for data science, AI, and ML professionals.

From Norm to Orthogonality: Fundamental Mathematics for Machine Learning with Intuitive Examples…

For understanding the mathematics for machine learning algorithms, especially deep learning algorithms, it is essential to build up the…

For understanding the mathematics for machine learning algorithms, especially deep learning algorithms, it is essential to build up the mathematical concepts from foundational to more advanced. Unfortunately, Mathematical theories are too hard/abstract/dry to digest in many cases. Imagine you are eating a pizza, it is always easier and more fun to go with a coke.

The purpose of this article is to provide intuitive examples for fundamental mathematical theories to make the learning experience more enjoyable and memorable, which is to serve chicken wings with beer, fries with ketchup, and rib-eye with wine.

The 3-course fundamental mathematics for machine learning meal is organized as follows:

From Scalar to Tensor: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 1/3

  • What are Scalar, Vector, Matrix and Tensor?
  • Addition between Scalar, Vector, and Matrix
  • Multiplication between Scalar, Vector, and Matrix
  • Identity and inverse matrix
  • Diagonal matrix and symmetric matrix

From Norm to Orthogonality: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 2/3

  • 1-Norm, 2-Norm, Max Norm of Vectors
  • Orthogonal and orthonormal vectors
  • Orthogonal matrix

From Eigendecomposition to Determinant: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 3/3

  • Eigendecomposition of matrix: eigenvalue and eigenvector
  • The trace operator
  • Determinant of square matrix

In this article, we will go through the part 2/3, From Norm to Orthogonality with intuitive examples.

1-Norm, 2-Norm, max Norm of Vectors

How to measure the size of a vector? One approach is to use the norm function:

  • 1-Norm: in machine learning applications, it is commonly used when the difference between 0 and non-0 elements is important.

For example, the 1-Norm of vector v could be calculated as:

  • 2-Norm: known as the Euclidean norm, which is the Euclidean distance from origin to the point identified by vector x.
Photo credit to wikipedia
Photo credit to wikipedia

It is common to use the squared 2-Norm instead of 2-Norm itself to measure the size of a vector. The reason is that the squared 2-Norm can be calculated as:

which is more convenient than calculate 2-Norm itself. The following example shows how to calculate 2-Norm of vector v:

  • Max Norm: the largest absolute value of the element in the vector, which can be written as:

The following example shows the calculation of Max Norm of vector v:

Orthogonal and orthonormal vectors

A vector u and vector v are orthogonal to each other if and only if their dot product is 0:

For example, in 3-D Euclidean space,

In geometry, the two orthogonal vectors are mutually perpendicular in Euclidean space:

Photo credit to ubisafe
Photo credit to ubisafe

Vector u and vector v are a pair of orthonormal vectors means:

It can be expanded to following equations in 3-D Euclidean space:

For example,

Therefore, we say that, vector u and vector v are orthonormal.

Orthogonal matrix

An orthogonal matrix is a square matrix whose rows and columns are orthonormal:

For example, the following matrix is orthogonal because:

This implies that a matrix is orthogonal if its transpose equal to its inverse:

Therefore, orthogonal matrix is of interest in Machine Learning because the inverse of matrix is very cheap to compute. The thing we need to pay attention is that the rows and columns in orthogonal matrices are not merely orthogonal but also orthonormal.

Congratulations! You have finished two-third of Fundamental Mathematics for Machine Learning with Intuitive Examples. You can make it!

Next step: From Eigendecomposition to Determinant: Fundamental Mathematics for Machine Learning with Intuitive Examples Part 3/3


Related Articles