Encoding may be seen as the art of abstracting the most salient patterns from reality. Such abstractions are indispensable for the purpose of generalization. For instance, consider the textbook example of recognizing cats in pictures. A trained neural network should be able to detect cats even in previously unseen images. This is only possible if the network can extrapolate its training observations.
Under the hood, the network extracts features such as the shape of the ears, the length of the body, etc. It reduces the cat to a series of shapes and attributes, which can be used to spot furry felines in new photos as well. In other word, the network preserves sufficient information to generalize the concept of a cat, while tossing all details and variations that obscure it.
Although art and machine learning have little in common at surface level – at least, until Dall·E 2 and other art generators entered the AI domain – it appears contemporary data scientists and the abstract artists of old have quite similar ambitions!
Introduction
So what’s the deal of this article? Frankly, the ambitions are fairly modest:
- Help to visualize what a ML algorithms do at an intuitive level
- Illustrate a few links between human and machinal attempts to abstract and reconstruct reality
- Perhaps trigger some thought among ML practitioners?
Disclaimers: There are many people who know much more about art than I do, so I won’t pretend to offer a full comprehension of art history. Furthermore, the article is anecdotal rather than the product of any formal study, and focuses only on those points where ML and art appear to overlap. For completeness, I should mention the article focuses on European art, looking back no more than a few hundred years, and highlights two movements somewhat arbitrarily. Similarly, I also don’t want to get too bogged down in precise descriptions of decoders and clustering algorithms, so ‘Machine Learning’ will be interpreted rather loosely.
Enough disclaimers, let’s get started.
Impressionism – Loosening ties with reality
For many years, artists strived towards realistic and accurate representations of the world, getting every reflection, water drop and wrinkle correct to the tiniest detail. Ultimately, artists started to deviate from this dogma, as seen in movements such as Impressionism (led by painters like Claude Monet and Pierre-Auguste Renoir).
Rather than perfectly mirroring observations, impressionists aimed to render their interpretation of reality as observed on the spot. They used quick and relatively crude brush strokes to capture the moment, especially the fickle light. The outdoor scene below remains easily identifiable, yet uses substantially less ‘information’ to transmit the real-world image.
In machine learning terms – the artist uses substantially less information/data points/features to represent the scene. See for instance the link with Image Segmentation, here using the k-means algorithm to cluster neighboring pixels. It creates larger shapes to be represented by a single color (i.e., data point). With much less data, we represent the same scene, although inevitably at a loss of granularity.
De Stijl— A quest for abstraction
Let’s continue our abstraction journey. De Stijl (Dutch for ‘The Style’) was a group of primarily Dutch painters, with Piet Mondriaan and Gerrit Rietveld arguably being the most widely known. They were known for their push towards absolute abstraction of art.
Although to some their art may invoke statements like ‘my five-year old daughter could do that’, each work reflects deep critical thought on how to represent the world with the bare minimum of visual means. Colors – if bearing any relevance at all – are merely combinations of primary colors, orthogonal lines embodiments of dynamic tensions. See for instance how Mondriaan reflected on one of his paintings:
"If the masc. [masculine] is the vertic. [vertical] line, then a man will recognize this element in the rising line of a forest; in the horizont. [horizontal] lines of the sea he will see his complement. Woman, with the horizont. line as element, sees herself in the recumbent lines of the sea, and her complement in the vert. lines of the forest." – Piet Mondriaan, 1912 [Mondrian, – The Art of Destruction]
Before we get at that level of abstraction, let’s take a moment to consider the work ‘De Storm’ of Bart van der Leck below. Compared to the real-world scene it might depict, it contains very little detail. Only a minimum of information is transmitted – a large yellow plane for the beach, a blue one for the sea, a shape that can be interpreted as a large wave. From this, we might deduce the setting – two women walking on the beach, looking out on a stormy sea.
Without the description, would you get all that though? Would you derive this is a beach, would you note the heavy wind gusts? Or has the painting – in all its abstractions – lost too much information already?
Another example?
The painting below (also Van der Leck) uses a fairly minimal amount of information. Still, we can identify a woman, child and a plane, event though only working with geometric shapes. It seems we need only little information to capture the essence of world, although you might argue the meaning of the scene is lost.
Time to dive even deeper into the abstraction. Like many image recognizers in Machine Learning, Theo van Doesburg completely omit colors from his work, focusing solely on shapes and patterns.
The study below illustrates that De Stijl did not blindly draw some lines on a canvas – traces of the original city view are still visible in the final work. Without the description tag, I doubt anyone could derive this painting represents the city of Utrecht though!
This is what we see when our model is not sufficiently powerful to capture all relevant patterns, e.g., a neural network with insufficient layers or nodes, or a linear model trying to capture non-linear patterns. The model might capture some features of the input, but the encoding is of insufficient quality to properly reconstruct the original observation.
Let’s check a Mondriaan painting now. Without the accompanying sign, can you deduce what it represents? Would a decoder be able to reconstruct the original inspiration for this work?
Although barely interpretable as a representation of reality, the members of De Stijl still sought more – an abstraction that no longer represents nature itself. Let’s see a final painting.
Things get increasingly fuzzy, as the artist does not even attempt to represent a natural observation. Instead, the painting seeks to reprentent the elementary building blocks of the universe, relying solely on the mind. The rectangles are uncentered, asymmetric and dynamic, yet co-exist in a certain harmony. The primary colors suffice to recreate the entire spectrum of all that is visible to our eyes. The unbounded edges imply the work expand indefinitely beyond the painting. The essence of the entire universe, encoded into a single canvas of rectangles.
Try Decoding that.
Closing words
Machine Learning seeks to subtract generalizable patterns from reality, chipping away at the details and noise until only the essence is preserved. Similarly, abstract artists as discussed in this article seek to strip down nature, until only the core truth remains.
In both cases, the question is whether we – be it human beings or a decoding algorithm – can reconstruct the original observation from the encoded representation. Encodings are highly effective in reducing dimensions or data required. However, if encodings are pushed too far, essential information is lost and can no longer be retrieved.
As machine learning algorithms are often perceived as a black box, the visual abstractions in this article may help in grasping what such algorithms attempt to do, at least at an intuitive level. Artists show that a ton of information may be removed from an observation, without losing the crucial patterns. At the same time, the correct interpretation relies increasingly on the power of the decoder.
Next time your ML algorithm struggles to learn anything useful, you might want to remember the artists of De Stijl and their quest to transcend the natural world. Perhaps, your algorithm simply went a bit too far down the path of abstraction.
Enjoyed the article? You may also like the following ones:
Is "Small Data" The Next Big Thing In Data Science?
Zen and the Art of Data Science – How To Avoid Gumption Traps
References
Composition avec grand plan rouge, jaune, noir, gris et bleu – Wikipédia
