The world’s leading publication for data science, AI, and ML professionals.

How AI Could Help Preserve Art

Art masterpieces are a risk at any time; AI and new technologies can give a hand

images generated by the author using OpenAI's DALL-E 2
images generated by the author using OpenAI’s DALL-E 2

In recent months there has been talk about how artificial intelligence can create images from textual prompts. Therefore, when one associates the words artificial intelligence and Art, one immediately thinks of DALL-E, Stable Diffusion, and other algorithms. In this article, instead, I want to discuss why artworks are often less safe than we think, and how artificial intelligence can help preserve them.

In hatred of beauty: What threatens the memory of the world?

The Louvre museum in ruins as imagined by the AI. image created by the author using Stable Diffusion
The Louvre museum in ruins as imagined by the AI. image created by the author using Stable Diffusion

"Every act of creation is first of all an act of destruction." ― Pablo Picasso"

It is a mistake to think that cultural heritage is safe. Many of humanity’s most valuable works are also among the most fragile. Throughout history, only a fraction of works of art has managed to survive over time.

For example, during wars, cultural heritage is often damaged. During antiquity, it was considered common practice to loot newly conquered territories, a tradition that was maintained during colonialism and Napoleonic plundering. In addition, during World War II a huge number of works of art were damaged or lost forever. Several works were stolen by the Nazis (and never found again), while others were destroyed during the Allied-led bombing raids on Germany.

Even today, during the war in Syria, millennia-old cities such as Aleppo have been brutally destroyed (more than 70% of the city according to the UN). Not to mention the acts of terrorist groups that destroyed and looted Palmyra and Iraq’s museums (often many of these works were resold to buy weapons). In addition, dictatorial regimes have often destroyed important works of art even in recent times (Buddhas of Bamiyan destroyed by the Taliban).

Buddha statues were destroyed by the Taliban. Images source: here
Buddha statues were destroyed by the Taliban. Images source: here

In addition, many works were destroyed or damaged during natural events. Often events such as earthquakes or other natural disasters lead to the loss of valuable works. During the Florence Flood of November 4, 1966, thousands of precious and ancient manuscripts were covered with mud and heavily damaged (some are still being restored). Cimabue’s precious crucifix, a 14th-century work, was also affected by the flood and required delicate restoration. Even today, many areas are at risk of flooding, and the phenomenon of _acqua alta_ (high water) is a risk to Venice.

Images from the Florence flood. In the left panel, the flooded historic center. Middle panel, some volunteers help carry paintings from the Uffizi museum to safety. Right panel, volunteers shovel mud from the streets.. Image source: here
Images from the Florence flood. In the left panel, the flooded historic center. Middle panel, some volunteers help carry paintings from the Uffizi museum to safety. Right panel, volunteers shovel mud from the streets.. Image source: here
Cimabue's crucifix, left before flood damage, right after restoration. image source: here
Cimabue’s crucifix, left before flood damage, right after restoration. image source: here

During the earthquake that destroyed Lisbon in 1755, a great many valuable ancient volumes kept in the Royal Library were lost (along with works by Titian, Rubens, Correggio, and Caravaggio). In addition, many works were lost in the fire of the Royal Alcazar of Madrid in 1734 (priceless works by Velazquez, Rubens, Bosch, Brueghel, Van Dyck, El Greco, Leonardo da Vinci, Raphael, and many others). Even today, fires like the one at the National Museum of Brazil in 2018 or the one at Notre Dame Cathedral in 2019 demonstrate how such destructive events can still happen.

In addition, some natural phenomena that cause damage to artworks are caused by human activities. In fact, pollution and climate change put artistic and architectural heritage at risk. For example, acid rain is accelerating the erosion of the Egyptian Sphinx, but it is also a serious problem for marble buildings. Rising temperatures are thought to be catalyzing chemical reactions by increasing damage to historic buildings.

The left panel shows Notre Dame Cathedral on fire in 2019. Middle panel, the Brazil Museum on fire. Right panel, Harvard University wraps some of the bronze and marble statues on its campus with waterproof covers each winter to protect them from corrosion caused by acid rain and acid snow. Image assembled by the author using images from Wikipedia
The left panel shows Notre Dame Cathedral on fire in 2019. Middle panel, the Brazil Museum on fire. Right panel, Harvard University wraps some of the bronze and marble statues on its campus with waterproof covers each winter to protect them from corrosion caused by acid rain and acid snow. Image assembled by the author using images from Wikipedia

There are also accidental events in which artwork was lost (airplane crashes and other forms of transportation). In 2006, a man fell after stepping on his loose shoelace at the Fitzwilliam Museum (Cambridge) destroying three 17th-century Chinese vases. While in 2010, a woman fell on a Picasso painting at the Metropolitan Museum damaging it (a painting valued at $130 million and considered one of his masterpieces). In 2000, a Sotheby’s employee disposed of a box using a crushing machine, only the box was not empty but contained a Lucian Freud painting.

In addition, the neglect of artworks is also a serious problem. Frescoes, ancient paintings, statues, and monuments are fragile works, so maintenance and restoration are expensive and delicate operations. In 2012, the restoration of a painting by an amateur restorer in Spain caused a stir (but there are other examples such as the "potato head" of Palencia).

The first panel, Picasso's Le Rêve damaged during an auction. The second panel is Picasso's painting "The actor" which was damaged at New York's MET by a falling woman. The third and fourth panels are the painting Ecce Home painted by Elías García Martínez, and the result of the restoration in 2012. Image assembled by the author using images from Wikipedia
The first panel, Picasso’s Le Rêve damaged during an auction. The second panel is Picasso’s painting "The actor" which was damaged at New York’s MET by a falling woman. The third and fourth panels are the painting Ecce Home painted by Elías García Martínez, and the result of the restoration in 2012. Image assembled by the author using images from Wikipedia

In addition, we could also add neglect (such as the collapses caused in Pompeii), vandals and madmen (such as the man who damaged Michelangelo’s Pietà), art trafficking, economical interest (for example, when building dams), and other causes as well. That is why UNESCO, which reviewed the World Heritage Sites, also keeps a less prestigious list where the heritage sites that are at risk are included.

The destruction of a work of art goes beyond the mere economic value of the work. When it was done intentionally in both ancient and modern times it was to erase the memory of a people (whether religious or cultural). In recent times, these same mechanisms have been used to destroy archaeological remains in Bosnia, Syria, and Afghanistan (considered contrary to religious doctrine). Moreover, the debate is more topical than ever as, for example, those who call for the return of goods stolen during colonialism (e.g., the famous Benin bronzes that represent the history of the nation and have been scattered in European museums) claim.

A people without the knowledge of their past history, origin and culture is like a tree without roots. – Marcus Garvey

As we have seen, heritage is at risk from natural phenomena but also from political choices when funds earmarked for preservation are cut. There is a civic duty in all of us in protecting what is our memory and this, in my opinion, also extends to data science. In fact, the use of Artificial Intelligence is becoming more and more democratized and can be used at less cost by anyone for social applications.

left panel: one of the Benin bronzes, now in the British museum and vehemently requested to be returned. Middle panel, part of the ceremonial beard of the Sphinx of Giza and now in the British Museum. Right panel, Parthenon marbles that Greece is asking to have returned.
left panel: one of the Benin bronzes, now in the British museum and vehemently requested to be returned. Middle panel, part of the ceremonial beard of the Sphinx of Giza and now in the British Museum. Right panel, Parthenon marbles that Greece is asking to have returned.

In short, artistic works are fragile and often more at risk than people think. How can science and artificial intelligence protect them?

Artificial intelligence to save human creativity

images generated by the author using OpenAI's DALL-E 2
images generated by the author using OpenAI’s DALL-E 2

To begin with, new scientific investigation techniques allow us to learn about the works. Even the greatest artists begin their works with a sketch and often rethink as they go. Today we have several techniques that allow us to analyze pictorial works (such as x-rays), which are not only noninvasive but allow us to tell the story of the work. However, these techniques produce data that are often difficult to interpret (especially with several overlapping images), so Machine Learning algorithms have been developed for image analysis.

X-ray exposure is able to show underdrawings or changes in progress. For example, this showed that Rembrandt fine-tuned the composition of the figures in his masterpiece Syndics of the Drapers’ Guild several times. Leonardo da Vinci himself had drawn angels and other figures before painting The Virgin of the Rocks. Although it can sometimes be easy to be able to identify the artist’s various interventions, there were often several patterns that the artist repainted several times generating several overlapping images. These patterns are difficult to distinguish, and AI helps to be able to reconstruct the different phases of the work.

Rembrandt's painting and its radiography. Image assembled by the author using images from Wikipedia
Rembrandt’s painting and its radiography. Image assembled by the author using images from Wikipedia

Artificial intelligence has also proven to be useful in restoration. For example, it has been successfully used in digital restoration (photography, articles, and even manuscripts). The MACH laboratory in Cambridge has used AI algorithms to identify damage and virtually reconstruct the images in manuscripts (a process called inpainting). Similar technologies have been used to reconstruct damaged photographs, color black and white photographs, reconstruct the image of frescoes, and so on.

the panel above: Inpainting of large image region with large damaged areas. the image shows the damaged design (left), the inpainting domain (center), and the final result right. Image source: [here](https://en.wikipedia.org/wiki/Digital_photograph_restoration). bottom panel: digital restoration of a photograph (before and after restoration). image source: here
the panel above: Inpainting of large image region with large damaged areas. the image shows the damaged design (left), the inpainting domain (center), and the final result right. Image source: [here](https://en.wikipedia.org/wiki/Digital_photograph_restoration). bottom panel: digital restoration of a photograph (before and after restoration). image source: here

As an interesting example, researchers recently reconstructed with AI and projected how Rembrandt‘s masterpiece, The Night Watch, must have originally looked (the painting was arbitrarily shortened when it was moved to another site). Furthermore, such a technique can be used to reconstruct works considered lost: for example, two panels are missing from the famous Van Eyck brothers’ Ghent Altarpiece (1432) and researchers have used a convolutional neural network to try to faithfully reconstruct the two panels.

"even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be." – source

The reconstruction of lost paintings is still a controversial application. Indeed, when an attempt was made to recreate lost Klimt paintings with AI (in 1945, three of Klimt’s masterpieces were irretrievably lost), researchers made it clear that the idea was not to replace but to give an idea of what is considered lost forever.

The "Adoration of the Mystic Lamb" which is also called the _Ghent Altarpiece. image source: here_
The "Adoration of the Mystic Lamb" which is also called the _Ghent Altarpiece. image source: here_

Another intriguing use of artificial intelligence is the authentication of paintings by algorithms. Indeed, valuations of works are a huge market and often attributing works is not easy (especially if it is the work of a painter or someone in his workshop). Recently, a method has been proposed whereby by studying the topography of the work, the signature of an author can be reconstructed. In short, the surface height information is recorded (at 50 microns spatial resolution) and then passed through a convolutional neural network (CNN), so that differences in brush strokes can be studied.

"Many notable artists, including El Greco, Rembrandt, and Peter Paul Rubens, employed workshops, of varying sizes and structures, to meet market demands for their art. Hence, there is need for unbiased and quantitative methods to lend insight into disputed attributions of workshop paintings." – article’s author (source)

Similar methods could be very useful for avoiding counterfeits of works of art, even being useful for dating and attributing works. In addition, algorithms that can recognize the signature of work can be useful against artwork trafficking.

The data preparation and analysis workflow that is used to recognize the brush strokes style of an author. image source: original paper
The data preparation and analysis workflow that is used to recognize the brush strokes style of an author. image source: original paper

Artificial intelligence to dig deeper

Image by Vicky T at unsplah.com
Image by Vicky T at unsplah.com

Artificial intelligence and its applications will also have an impact on archaeology and archaeological heritage.

Moreover, X-Ray is not restricted to paintings. In fact, researchers often also analyze objects such as glass (for example, to understand the workmanship), mummies, and statues. In addition, other techniques such as CT scans are also often used. The Antikythera mechanism (mysterious Greek artifact) itself has been analyzed by X-ray to study its possible operation. In all these cases AI algorithms applied to image analysis have proven to be very useful.

Another interesting case is palimpsests: parchments or books that were erased by scraping off the ink and then rewritten again (parchment was expensive and was therefore reused by amanuensis monks). Today, erased text can be reconstructed using imaging techniques, allowing us to rediscover masterpieces of antiquity believed lost. Recently, using artificial intelligence and X-rays it has been possible to transcribe to decode the Archimedes Palimpsest (which contains two works of Archimedes thought to be lost).

left panel: Radiograph of the African Songye Power Figure at the Indianapolis Museum of Art. right panels: images of the Archimedes Palimpsest under different light sources to perform the analysis. Image assembled by the author using images from Wikipedia
left panel: Radiograph of the African Songye Power Figure at the Indianapolis Museum of Art. right panels: images of the Archimedes Palimpsest under different light sources to perform the analysis. Image assembled by the author using images from Wikipedia

The terrible eruption of 79 AD covered the cities of Pompeii and Herculaneum with a deluge of ash and lapilli. Beneath this blanket was found a valuable library of papyri (papyri rarely survive the Mediterranean climate). Unfortunately, previous attempts to unroll and decipher them had led to the destruction of the papyri. Fortunately, new imaging techniques make it possible to analyze them without the need to unroll them.

"Although it is possible to note that there is writing on every roll of the Herculaneum papyri, opening it would require the papyrus to have its own flexibility. And there is no more flexibility,"- Brant Sales (source)

The X-ray approach had been successfully attempted with a 1,700-year-old Hebrew parchment found in En-Gedi, Israel. Unfortunately, while the Israeli scroll contained a metal-based ink that showed well on x-rays, the Hercolanus papyri were written with a carbon-based ink, meaning that "there is no obvious contrast between the writing and the papyrus in x-ray scans". For this, the study authors used more energetic X-rays but also artificial intelligence. The authors used a 3D-convolutional neural network to detect the text and decipher it.

On the left: Herculaneum fragment results from the analysis. On the right: Overview of a system to train a neural network to detect carbon ink in CT scan data. (source: [here](https://creativecommons.org/licenses/by/4.0/), license: here)
On the left: Herculaneum fragment results from the analysis. On the right: Overview of a system to train a neural network to detect carbon ink in CT scan data. (source: [here](https://creativecommons.org/licenses/by/4.0/), license: here)

Image analysis can also be used to discover unknown archaeological sites. The discovery of an archaeological site is often a fortuitous event (excavation for other reasons) or requires expensive investigation. Indeed, lidar technology (in which one targets an object or a surface with a laser), has been successfully used to detect new archaeological sites (in Mexico the remains of Angamuco were thus found). This technology has also been used to reveal anthropogenic changes to the Angkor landscape. Lidar, thermal images, and satellite images can therefore be analyzed with AI to monitor the status of archaeological sites and study interventions.

In the past, these instruments were on helicopters or small planes. Today, however, drones are increasingly popular and allow greater flexibility. For example, one research project used drones to map the ruins of Pompeii. Mapping is the first step to verify which structures are at risk, follow the evolution of the site, and plan priorities for interventions. Furthermore, drones can also be used in difficult areas such as underwater archaeology.

As we have seen AI can be used to reconstruct damaged paintings. The same approach can also be used for archaeological artifacts such as mosaics. Recently, a paper was published that presents an interesting approach. The authors tested the "outpainting" capability of OpenAI’s DALL-E (the AI takes as input an incomplete image and fills in the missing parts). They tested them either with mosaics that were already damaged (both figurative and geometrically patterned) or by artificially removing parts of well-preserved mosaics, so they could compare the results.

The approach and results are intriguing; it also takes advantage of an existing algorithm by trying it out under different conditions. This demonstrates the flexibility and emergent properties of these algorithms. On the other hand, as the authors note, the results are not always exciting:

However, the reconstructions show several mistakes and therefore, it is still far in most cases from the quality of a manual reconstruction. The worst performance is obtained when recreating faces and in the presence of nudity (this is due to DALL-E policies on the contents of images). For geometric shapes, the performance seems to be better, but DALL-E has some limitations on the color recreation and for some of the forms, especially when they are small. – source: original article

Mosaic reconstruction using artificial intelligence. In the panel above: the mosaic of the Battle of the Amazons as in the original (left) and reconstructed with AI (right). In the panel below: the original mosaic (left) is artificially damaged (center) and reconstructed by the algorithm (left). Image assembled by the author, source: original article
Mosaic reconstruction using artificial intelligence. In the panel above: the mosaic of the Battle of the Amazons as in the original (left) and reconstructed with AI (right). In the panel below: the original mosaic (left) is artificially damaged (center) and reconstructed by the algorithm (left). Image assembled by the author, source: original article

Artificial intelligence can also be used to catalog and automate tedious tasks. Archaeologists find thousands of fragments of pottery (especially Roman) and it is tedious work to analyze thousands of fragments of vases, amphorae, and plates. On the other hand, all these fragments once cataloged and studied the relationships between them, can provide valuable information on the daily life of past civilizations. In Cambridge, they have developed an algorithm that matches the fragment with the pottery profile in a database. This approach makes it possible to quickly catalog, and then with other algorithms study, the distribution of the various types of ceramics in an archaeological site.

This approach is not limited to Roman ceramics. Researchers at the University of Arizona have used a similar approach to classify the designs and patterns of ancient Pueblo ceramics.

Archaeologists often find epigraphs, but these inscriptions are often damaged over the centuries and rendered illegible. Recently, DeepMind presented Ithaca (a follow-up of the previous model called Pythia), an artificial intelligence model capable of finding missing characters in a text that has been damaged. The authors of DeepMind trained their model on one of the largest corpora of Greek inscriptions to obtain results similar to those obtained by human epigraphers. Similar approaches have also been attempted with other languages such as the Scythian language, 3000-year-old Chinese oracle bone scripts, and Persian cuneiform tablets.

left panel: ancient Roman pottery. middle panel: the rosetta stela (stela that allowed the hieroglyphs to be deciphered). right panel: Chinese oracle bone script. Image assembled by the author using images from Wikipedia
left panel: ancient Roman pottery. middle panel: the rosetta stela (stela that allowed the hieroglyphs to be deciphered). right panel: Chinese oracle bone script. Image assembled by the author using images from Wikipedia

"Just because you can read the letters, that doesn’t mean you know what they mean" – Regina Barzilay (source)

Although it is not easy to decipher ancient inscriptions when the language is known (Greek, Latin, and so on), there are cases when the language is lost. Languages can be grouped into different families according to the traits they share (alphabets, vocabulary, grammar, sounds, and so on). Often languages share a common root ( e.g., Neo-Latin languages, "aquam" in Latin, "acqua" in Italian, "agua" in Spanish) and then undergo an evolutionary path that causes them to diverge. These principles have been used by linguists, to look for similarities and common patterns in order to decipher dead languages.

AI has been shown to be capable of finding patterns and finding similarities. Therefore, this approach has been attempted to try to decode lost languages such as Ugaritic or Linear B. In this case, the authors used a model based on LSTM and embedding and obtained some interesting results. The approach used, does nothing more than look for spans in the lost texts with known tokens.

In the panel above: a damaged stele in ancient Greek telling the story of the acropolis and to the right an example of an inscription reconstructed with Pythia (blue when correct, purple incorrect), source: original article. Bottom panel, an example of Linear B and Ugaritic writing, source: Wikipedia
In the panel above: a damaged stele in ancient Greek telling the story of the acropolis and to the right an example of an inscription reconstructed with Pythia (blue when correct, purple incorrect), source: original article. Bottom panel, an example of Linear B and Ugaritic writing, source: Wikipedia

Parting thoughts

images generated by the author using OpenAI's DALL-E 2
images generated by the author using OpenAI’s DALL-E 2

Recent years have seen a large effort to digitize art. Major museums (such as the MET)have created large databases of paintings, books, statues, objects, artifacts, and so on. Often these huge digital libraries are accessible to the public and scholars free of charge. The institutions themselves are moving towards coordinated initiatives. For instance, the European Union has created guidelines to digitize the immense cultural heritage.

These initiatives are aimed at democratizing access to collections and cultural heritage. Indeed, many works preserved in museums can be seen by the public simply by visiting the sites. Moreover, museums only display a small part of their collections (many works are stored in warehouses and are never exhibited except in rare exhibitions) so these initiatives allow access to works that normally could not be seen. On the one hand, these initiatives allow valuable resources for scholars who want to study the works. On the other hand, algorithms need data and this allows increasingly sophisticated artificial intelligence models to be trained.

As we have seen, works, however, preserved in a museum, are delicate objects and can be lost at any time. Safeguarding them is an important and difficult job, whether it is to preserve them or restore them. In these tasks, artificial intelligence can help (monitoring, studying them, and so on). On the other hand, technology needs to be backed by policies and investments to preserve heritage.

image created by the author using Stable Diffusion
image created by the author using Stable Diffusion

Not least, AI has developed at great speed in recent years and interesting perspectives open up for the increasingly powerful models. In fact, as we have seen many of the models used are convolutional networks that have proven to be efficient for tasks involving images. However, all other areas of machine learning can be used as well. In fact, for example, unsupervised clustering can be used to group ceramic fragments.

In addition, the first models that were used to reconstruct inscriptions and translate dead languages were based on encoder-decoders and LSTMs. Pythia also used the same architecture, but already the later DeepMind Ithaca article was based on a transformer-like architecture. As vision transformers are shown to be effective for imaging we can expect more similar models in the coming years.

Another note is how it is not necessarily necessary to develop new and sophisticated algorithms for new tasks: as we have seen some researchers have leveraged DALL-E which is publicly accessible via a website. This shows that many of the available algorithms could be repurposed in the future.

Aside from technical demonstrations of AI capabilities, it opens up real perspectives and impact. For example, having a good digital image is the first step in restoring a painting or mosaic. Also, many archaeological sites are understaffed and surveillance is expensive and problematic. Not to mention that AI will be able to help make decisions, prioritize interventions, and reduce costs.

"These things look like cartoons. They don’t look like Klimt paintings. It’s like people who try to clone their dogs. You can do it, but it’s not the same dog." – Jane Kallir about AI reconstruction of lost Klimt paintings (source)

Open ethical questions certainly remain. The result as seen with the mosaics should be taken carefully. After all, the model sometimes more than reconstructing history seems to be rewriting it. As in the case of Rembrandt’s painting, the algorithm tries to guess based on the data at its disposal, and in fact, the curators of the exhibition pointed out the separation between the original part and what has been reconstructed.

In addition, satellite imagery and LIDAR technology have identified thousands of new archaeological sites not yet explored. These may be excavated by grave robbers long before archaeologists can work on them. As important as it is to consider publishing the data, one must also move cautiously.

Another thorny case is restorations. Since the 19th century, every new technology has been exploited in restoration without considering the possible damage. For example, concrete has burdened structures causing sometimes more damage than expected.

If you have found it interesting:

You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support!

Here is the link to my GitHub repository, where I am planning to collect code and many resources related to machine learning, artificial intelligence, and more.

GitHub – SalvatoreRa/tutorial: Tutorials on machine learning, artificial intelligence, data science…

Or feel free to check out some of my other articles on Medium:

How artificial intelligence could save the Amazon rainforest

Reimagining The Little Prince with AI

Speaking the Language of Life: How AlphaFold2 and Co. Are Changing Biology

Blending the power of AI with the delicacy of poetry


Related Articles