Introduction to Geometric Deep Learning

Geometric Deep Learning

A series of blog posts, on Geometric Deep Learning (GDL) Course, at AMMI program; African Master’s of Machine Intelligence, taught by Michael Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković.

The rapid development of deep learning has created different neural network architectures that have shown success in various data science fields. At the same time, we don’t have a clear unifying principles between these architectures, so it’s difficult to understand the relationships between them. In this post, we review the definition of the word symmetry among various mathematicians touching on some historical context, the appearance of the Erlangen Programme, and how it came to deep learning in the name Geometric Deep Learning. We will see how the current state of deep learning reminds us of many geometries in the 19th century.

Our main references are the GDL proto-book by the four instructors, and the GDL Course at AMMI.

This blog post was co-authored with MohammedElfatih Salah

If you want a simple word to describe what we are going to talk about, and get an intuition about the idea of geometric deep learning (GDL), it will be symmetry. Let’s go back in history and see how the mathematicians define this word. One of the first definitions of symmetry came by H. Weyl, one of the best mathematicians in the 20th century. Weyl said in his words about symmetry: ​​

Going a little back to ancient Greeks who used this term symmetry also to express the beauty of proportion, harmony, and music vaguely.

The first example is the five regular polyhedra, or the Platonic solids proposed by the Athenian philosopher Plato. A Platonic solid is a convex regular polyhedron in three-dimensional space. In a regular polyhedron, the faces are identical in shape and size, as well as all angles and all edges. And also the same number of faces meet at each vertex.

Similar example from ancient Greeks, is the symmetric shape of water crystals came by the mathematician J. Kepler. He wrote an entire book called _On the six-cornered snowflake_ in which he explained the sixfold dihedral symmetry of snowflakes.

Euclidean geometry is also goes back to ancient Greeks. Euclid, another Greek mathematician, defined Euclidean geometry as follows:

"In a plane, given a line and a point not on it, at most one line parallel to the given line can be drawn through the point."

With the advent of the 19th century, and after the development of _projective geometry_; by the French J. V. Poncelet, this Euclidean geometry came to an end. In projective geometry, points and lines are interchangeable; There is no such thing as parallelism as any two lines at exactly one point.

Someone could asked what is the first appearance of non-Euclidean geometry. We would say it came from the Russian mathematician N. Lobachevsky. He stated that there is more than one line at any point that can be extended through that point and run parallel to another line of which that point is not a part. Similar idea also, but we will not explain here, is the _differential geometry_ of surfaces (Riemannian geometry) by the German mathematician B. Riemann.

After all these developments, and towards the end of the 19th century, the mathematicians debated which geometry is the right one and how it should be defined. Until the German mathematician F. Klein published the Erlangen Programme, in which he gave an answer to all these questions.

When F. Klein proposed the Erlangen Programme, he was only 23 years old, at this age he was appointed as a professor in the small Bavarian university. In this Erlangen Programme, Klein describes geometry as the study of invariants and symmetries; which means the properties that are unchanged under a certain class of transformations.

So we can see clearly from Klein’s words, that the appropriate choice of symmetry can define different geometries. For example, the symmetries that define Euclidean geometry are rigid motions. So these are translations, reflections, and rotations that preserve properties such as angles, distances, areas, the parallelism of lines, and intersections.

Another important things is that this Erlangen Programme extended to other fields, most notably in physics. A clear example here could be Noether’s theorem, which stated that each distinct symmetry of the functioning of a physical system has a corresponding protective law. Prior to this theorem, if you would like to discover the conservation of energy you have to do very detailed material experimental observations and measurements to see that the energy remains the same even for some minor errors. But Noether’s theorem says that the conservation of energy arises from the translational symmetry of time. So it is relatively clear that the results of your experiment would be the same if you did it yesterday, today, or if you did it tomorrow.

We can also mention the words by P. Anderson, one of the Nobel-winning physicists: "It is only slightly overstating the case to say that Physics is the study of symmetry."

After this long historical context about geometry and symmetry, you may ask what this has to do with deep learning. The simple answer is that the current state of deep learning reminds us of many geometries in the 19th century. Since we have different neural network architectures for different types of data (such as CNNs for special data, transformers for sequential data, and RNNs for temporal data), but there are no clear unifying principles between these methods. Therefore, it’s difficult to understand the relations between them, which leads to the reinvention of the same concepts. Geometric deep learning reveals the basic principles that are unified behind all these architectures. We want to do this unification in the essence of the Erlangen Programme, which will be clarified in later posts.

Basically, the normal question will be why to study geometric deep learning, or what are the goals of GDL. We could say GDL serves two purposes. First, it explains why current state-of-the-art architectures are successful (because they respect invariance in the data which we will explain in a later post). Second, it proposes the main principles that if we follow them we can build better architecture for future tasks (we will also have another post on the GDL Blueprint).

The term ‘Geometric Deep Learning’ was popularized in a 2017 paper: Geometric Deep Learning: Going beyond Euclidean data in the IEEE Signal Processing Magazine authored by Bronstein et al. Most recently, Bronstein, Bruna, Cohen, and Veličković wrote a long article that is a preview of a book called Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.

In this series of posts we will explain in detail how by the two properties Symmetry and Scale Separation we can develop a GDL Blueprint that can serve as a framework for current state-of-the-art architectures. We will discuss the so-called Geometric Domains or the 5 Gs which include Grids, Groups, Graphs, Geodesics, and Gauges, and their appropriate structure, in the pipeline of the GDL Blueprint.

References:

• GDL Course, (AMMI, Summer 2021).
• Lecture 1 by M. M. Bronstein [ Video | Slides ].
• M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges (2021).

We thank Rami Ahmed for his helpful comments on the draft.