Vector Norms, Explained
Shedding light on a commonly misunderstood topic in linear algebra
Vector norms are extremely important in certain fields of engineering and mathematics. However, I think the education system often presents norms in a formulaic, “here’s how to calculate it” way as opposed to presenting an intuitive understanding of vector norms. Let’s see if we can remedy that situation with a more pragmatic discussion of vector norms.
What is a norm?
A norm is simply a way to describe the size of a vector, but the definition is often presented as
The mathematical definition seems complicated at first, but it is really just saying that a norm is some function that has certain properties. Let’s simply the above definition a bit to make it more intuitive.
A norm is a function from a real or complex vector space that has the following properties:
- Positive Definiteness: it is only zero at the origin
- Homogeneity: it commutes with scaling (i.e. we can pull out any scalars)
- Triangle Inequality: it obeys the triangle inequality so that the function acting on the sum of x and y is less than or equal to the sum of the functions acting on each element separately.
And there you have it! An algebraic understanding of norms. However, let’s add a bit more complexity. A norm isn’t just a single function. It can be many functions!
Types of Norms
Norms are abundant in mathematical fields. In fact, I bet many of you have seen a norm before and you didn’t even know it. Let’s discuss a few types of norms.
1-Norm
The 1-Norm, or L1 norm, is defined as
which is just a fancy way of the 1-Norm is the column sum of the absolute value of each entry. For Nx1 vectors, simply add the absolute value of each element and it will yield the 1-Norm.
You typically see 1-Norms used in machine learning applications. It can be used as a technique to reduce model coefficients, thus, reducing the overall complexity of the model. It can be used where sparsity of a vector is important as the 1-Norm is only influenced by the non-zero elements.
2-Norm
The 2-Norm, or L2 norm, is defined as
It is commonly referred to as the Euclidian norm. You have probably seen the 2-Norm as the standard way of calculating the length, or magnitude, of a vector. It is the “square root of the sum of squares” method. Engineers and scientists typically default to the Euclidian norm because distances behave according to this norm in nature for standard scales.
The most common application of Euclidian norms is using it to determine a vector’s magnitude and, therefore, distance from point A to point B.
P-Norm
The 1-Norm and the 2-Norm are P-Norms, where P=1 and P=2, respectively. We choose the values of one and two because they are commonly used throughout applications, but P can be set to any number greater than one. The P-Norm is defined as
Did you know: p must be greater than or equal to 1 because values greater than zero, but less than 1, violate the triangle inequality and, therefore, are not norms.
Values of P are often specific to the application. There are a few relatively popular ones that you should be aware of for a given field and/or application. However, since it is application-specific, I will leave it to you to explore the best P value for your application.
∞-Norm
The ∞-Norm is defined as,
which is the maximum absolute value of the rows. For Nx1 vectors, the ∞-Norm is simply the maximum absolute value in the vector since there are N rows.
One application of ∞-Norms is error analysis of numerical methods. The ∞-Norm provides a maximum bound which makes it the perfect estimator for error bounds.
Geometric Interpretation
Okay, so we know what a vector norm is and we know the different types of vector norms. For me, I didn’t gain an intuitive understanding of norms until I saw a geometric interpretation of what a norm is.
We have the geometric interpretation of the norms described above. The shapes above are unit shapes; that is, they cross the axes one unit away. Starting at the diamond (P=1), P can either increase or decrease. If it decreases, the sides of the diamond collapse inward toward the origin. If P increases, the sides begin to bend outward as seen in P=2. As P continues to increase, the sides continue outward, straightening until we get to P=∞, where it ends at a square.
It is easy to see that each type of norm behaves differently in the same vector space. You can see this algebraically, but the geometric interpretation helps solidify the concept. Therefore, different vector norms are used in different applications to produce a desired result.
Norms and Python
Let’s look at some code! We will review how to compute a norm using a common python library known as scipy; specifically, the linear algebra package. We will look at how to compute the norms listed above.
import scipy.linalg as la
import numpy as nparr = np.array([4, 3, 1, 25, 0, 5, 2, 4])
one_norm = la.norm(arr, ord=1)
two_norm = la.norm(arr, ord=2)
seven_norm = la.norm(arr, ord=7)
inf_norm = la.norm(arr, ord=np.inf)
Computation of a norm is made easy in the scipy library. We simply declare our vector and call the “norm” function. It is imperative that you specify which norm you want to compute as the default is the Euclidian norm (i.e. P=2). Let’s run the above code.
NOTE: P=7 is simply an illustrative example. I am not aware of any relevance in this value.
As we can see, the output is as expected. Notice that as we increase P, we converge to the value of the ∞-Norm. It is often much easier to compute norms via software rather than by hand. I often use python in error analysis for iterative techniques.
Conclusion
Vector norms are extremely common in mathematics and engineering. Much of the discussion here can be extrapolated beyond vectors and onto induced matrix norms. Hopefully, this helped give you a more intuitive understanding of matrix norms and how we can use software to compute them.
