
In recent months there has been talk about how artificial intelligence can create images from textual prompts. Therefore, when one associates the words artificial intelligence and Art, one immediately thinks of DALL-E, Stable Diffusion, and other algorithms. In this article, instead, I want to discuss why artworks are often less safe than we think, and how artificial intelligence can help preserve them.
In hatred of beauty: What threatens the memory of the world?

"Every act of creation is first of all an act of destruction." ― Pablo Picasso"
It is a mistake to think that cultural heritage is safe. Many of humanity’s most valuable works are also among the most fragile. Throughout history, only a fraction of works of art has managed to survive over time.
For example, during wars, cultural heritage is often damaged. During antiquity, it was considered common practice to loot newly conquered territories, a tradition that was maintained during colonialism and Napoleonic plundering. In addition, during World War II a huge number of works of art were damaged or lost forever. Several works were stolen by the Nazis (and never found again), while others were destroyed during the Allied-led bombing raids on Germany.
Even today, during the war in Syria, millennia-old cities such as Aleppo have been brutally destroyed (more than 70% of the city according to the UN). Not to mention the acts of terrorist groups that destroyed and looted Palmyra and Iraq’s museums (often many of these works were resold to buy weapons). In addition, dictatorial regimes have often destroyed important works of art even in recent times (Buddhas of Bamiyan destroyed by the Taliban).

In addition, many works were destroyed or damaged during natural events. Often events such as earthquakes or other natural disasters lead to the loss of valuable works. During the Florence Flood of November 4, 1966, thousands of precious and ancient manuscripts were covered with mud and heavily damaged (some are still being restored). Cimabue’s precious crucifix, a 14th-century work, was also affected by the flood and required delicate restoration. Even today, many areas are at risk of flooding, and the phenomenon of _acqua alta_ (high water) is a risk to Venice.


During the earthquake that destroyed Lisbon in 1755, a great many valuable ancient volumes kept in the Royal Library were lost (along with works by Titian, Rubens, Correggio, and Caravaggio). In addition, many works were lost in the fire of the Royal Alcazar of Madrid in 1734 (priceless works by Velazquez, Rubens, Bosch, Brueghel, Van Dyck, El Greco, Leonardo da Vinci, Raphael, and many others). Even today, fires like the one at the National Museum of Brazil in 2018 or the one at Notre Dame Cathedral in 2019 demonstrate how such destructive events can still happen.
In addition, some natural phenomena that cause damage to artworks are caused by human activities. In fact, pollution and climate change put artistic and architectural heritage at risk. For example, acid rain is accelerating the erosion of the Egyptian Sphinx, but it is also a serious problem for marble buildings. Rising temperatures are thought to be catalyzing chemical reactions by increasing damage to historic buildings.

There are also accidental events in which artwork was lost (airplane crashes and other forms of transportation). In 2006, a man fell after stepping on his loose shoelace at the Fitzwilliam Museum (Cambridge) destroying three 17th-century Chinese vases. While in 2010, a woman fell on a Picasso painting at the Metropolitan Museum damaging it (a painting valued at $130 million and considered one of his masterpieces). In 2000, a Sotheby’s employee disposed of a box using a crushing machine, only the box was not empty but contained a Lucian Freud painting.
In addition, the neglect of artworks is also a serious problem. Frescoes, ancient paintings, statues, and monuments are fragile works, so maintenance and restoration are expensive and delicate operations. In 2012, the restoration of a painting by an amateur restorer in Spain caused a stir (but there are other examples such as the "potato head" of Palencia).

In addition, we could also add neglect (such as the collapses caused in Pompeii), vandals and madmen (such as the man who damaged Michelangelo’s Pietà), art trafficking, economical interest (for example, when building dams), and other causes as well. That is why UNESCO, which reviewed the World Heritage Sites, also keeps a less prestigious list where the heritage sites that are at risk are included.
The destruction of a work of art goes beyond the mere economic value of the work. When it was done intentionally in both ancient and modern times it was to erase the memory of a people (whether religious or cultural). In recent times, these same mechanisms have been used to destroy archaeological remains in Bosnia, Syria, and Afghanistan (considered contrary to religious doctrine). Moreover, the debate is more topical than ever as, for example, those who call for the return of goods stolen during colonialism (e.g., the famous Benin bronzes that represent the history of the nation and have been scattered in European museums) claim.
A people without the knowledge of their past history, origin and culture is like a tree without roots. – Marcus Garvey
As we have seen, heritage is at risk from natural phenomena but also from political choices when funds earmarked for preservation are cut. There is a civic duty in all of us in protecting what is our memory and this, in my opinion, also extends to data science. In fact, the use of Artificial Intelligence is becoming more and more democratized and can be used at less cost by anyone for social applications.

In short, artistic works are fragile and often more at risk than people think. How can science and artificial intelligence protect them?
Artificial intelligence to save human creativity

To begin with, new scientific investigation techniques allow us to learn about the works. Even the greatest artists begin their works with a sketch and often rethink as they go. Today we have several techniques that allow us to analyze pictorial works (such as x-rays), which are not only noninvasive but allow us to tell the story of the work. However, these techniques produce data that are often difficult to interpret (especially with several overlapping images), so Machine Learning algorithms have been developed for image analysis.
X-ray exposure is able to show underdrawings or changes in progress. For example, this showed that Rembrandt fine-tuned the composition of the figures in his masterpiece Syndics of the Drapers’ Guild several times. Leonardo da Vinci himself had drawn angels and other figures before painting The Virgin of the Rocks. Although it can sometimes be easy to be able to identify the artist’s various interventions, there were often several patterns that the artist repainted several times generating several overlapping images. These patterns are difficult to distinguish, and AI helps to be able to reconstruct the different phases of the work.

Artificial intelligence has also proven to be useful in restoration. For example, it has been successfully used in digital restoration (photography, articles, and even manuscripts). The MACH laboratory in Cambridge has used AI algorithms to identify damage and virtually reconstruct the images in manuscripts (a process called inpainting). Similar technologies have been used to reconstruct damaged photographs, color black and white photographs, reconstruct the image of frescoes, and so on.
. bottom panel: digital restoration of a photograph (before and after restoration). image source: here](https://towardsdatascience.com/wp-content/uploads/2022/10/1IduiewyTLH4E4dltCw16UQ.png)
As an interesting example, researchers recently reconstructed with AI and projected how Rembrandt‘s masterpiece, The Night Watch, must have originally looked (the painting was arbitrarily shortened when it was moved to another site). Furthermore, such a technique can be used to reconstruct works considered lost: for example, two panels are missing from the famous Van Eyck brothers’ Ghent Altarpiece (1432) and researchers have used a convolutional neural network to try to faithfully reconstruct the two panels.
"even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence at the place where it happens to be." – source
The reconstruction of lost paintings is still a controversial application. Indeed, when an attempt was made to recreate lost Klimt paintings with AI (in 1945, three of Klimt’s masterpieces were irretrievably lost), researchers made it clear that the idea was not to replace but to give an idea of what is considered lost forever.

Another intriguing use of artificial intelligence is the authentication of paintings by algorithms. Indeed, valuations of works are a huge market and often attributing works is not easy (especially if it is the work of a painter or someone in his workshop). Recently, a method has been proposed whereby by studying the topography of the work, the signature of an author can be reconstructed. In short, the surface height information is recorded (at 50 microns spatial resolution) and then passed through a convolutional neural network (CNN), so that differences in brush strokes can be studied.
"Many notable artists, including El Greco, Rembrandt, and Peter Paul Rubens, employed workshops, of varying sizes and structures, to meet market demands for their art. Hence, there is need for unbiased and quantitative methods to lend insight into disputed attributions of workshop paintings." – article’s author (source)
Similar methods could be very useful for avoiding counterfeits of works of art, even being useful for dating and attributing works. In addition, algorithms that can recognize the signature of work can be useful against artwork trafficking.

Artificial intelligence to dig deeper

Artificial intelligence and its applications will also have an impact on archaeology and archaeological heritage.
Moreover, X-Ray is not restricted to paintings. In fact, researchers often also analyze objects such as glass (for example, to understand the workmanship), mummies, and statues. In addition, other techniques such as CT scans are also often used. The Antikythera mechanism (mysterious Greek artifact) itself has been analyzed by X-ray to study its possible operation. In all these cases AI algorithms applied to image analysis have proven to be very useful.
Another interesting case is palimpsests: parchments or books that were erased by scraping off the ink and then rewritten again (parchment was expensive and was therefore reused by amanuensis monks). Today, erased text can be reconstructed using imaging techniques, allowing us to rediscover masterpieces of antiquity believed lost. Recently, using artificial intelligence and X-rays it has been possible to transcribe to decode the Archimedes Palimpsest (which contains two works of Archimedes thought to be lost).

The terrible eruption of 79 AD covered the cities of Pompeii and Herculaneum with a deluge of ash and lapilli. Beneath this blanket was found a valuable library of papyri (papyri rarely survive the Mediterranean climate). Unfortunately, previous attempts to unroll and decipher them had led to the destruction of the papyri. Fortunately, new imaging techniques make it possible to analyze them without the need to unroll them.
"Although it is possible to note that there is writing on every roll of the Herculaneum papyri, opening it would require the papyrus to have its own flexibility. And there is no more flexibility,"- Brant Sales (source)
The X-ray approach had been successfully attempted with a 1,700-year-old Hebrew parchment found in En-Gedi, Israel. Unfortunately, while the Israeli scroll contained a metal-based ink that showed well on x-rays, the Hercolanus papyri were written with a carbon-based ink, meaning that "there is no obvious contrast between the writing and the papyrus in x-ray scans". For this, the study authors used more energetic X-rays but also artificial intelligence. The authors used a 3D-convolutional neural network to detect the text and decipher it.
, license: here)](https://towardsdatascience.com/wp-content/uploads/2022/10/1FxBznS6LPmKK09wVjbxyXg.png)
Image analysis can also be used to discover unknown archaeological sites. The discovery of an archaeological site is often a fortuitous event (excavation for other reasons) or requires expensive investigation. Indeed, lidar technology (in which one targets an object or a surface with a laser), has been successfully used to detect new archaeological sites (in Mexico the remains of Angamuco were thus found). This technology has also been used to reveal anthropogenic changes to the Angkor landscape. Lidar, thermal images, and satellite images can therefore be analyzed with AI to monitor the status of archaeological sites and study interventions.
In the past, these instruments were on helicopters or small planes. Today, however, drones are increasingly popular and allow greater flexibility. For example, one research project used drones to map the ruins of Pompeii. Mapping is the first step to verify which structures are at risk, follow the evolution of the site, and plan priorities for interventions. Furthermore, drones can also be used in difficult areas such as underwater archaeology.
As we have seen AI can be used to reconstruct damaged paintings. The same approach can also be used for archaeological artifacts such as mosaics. Recently, a paper was published that presents an interesting approach. The authors tested the "outpainting" capability of OpenAI’s DALL-E (the AI takes as input an incomplete image and fills in the missing parts). They tested them either with mosaics that were already damaged (both figurative and geometrically patterned) or by artificially removing parts of well-preserved mosaics, so they could compare the results.
The approach and results are intriguing; it also takes advantage of an existing algorithm by trying it out under different conditions. This demonstrates the flexibility and emergent properties of these algorithms. On the other hand, as the authors note, the results are not always exciting:
However, the reconstructions show several mistakes and therefore, it is still far in most cases from the quality of a manual reconstruction. The worst performance is obtained when recreating faces and in the presence of nudity (this is due to DALL-E policies on the contents of images). For geometric shapes, the performance seems to be better, but DALL-E has some limitations on the color recreation and for some of the forms, especially when they are small. – source: original article

Artificial intelligence can also be used to catalog and automate tedious tasks. Archaeologists find thousands of fragments of pottery (especially Roman) and it is tedious work to analyze thousands of fragments of vases, amphorae, and plates. On the other hand, all these fragments once cataloged and studied the relationships between them, can provide valuable information on the daily life of past civilizations. In Cambridge, they have developed an algorithm that matches the fragment with the pottery profile in a database. This approach makes it possible to quickly catalog, and then with other algorithms study, the distribution of the various types of ceramics in an archaeological site.
This approach is not limited to Roman ceramics. Researchers at the University of Arizona have used a similar approach to classify the designs and patterns of ancient Pueblo ceramics.
Archaeologists often find epigraphs, but these inscriptions are often damaged over the centuries and rendered illegible. Recently, DeepMind presented Ithaca (a follow-up of the previous model called Pythia), an artificial intelligence model capable of finding missing characters in a text that has been damaged. The authors of DeepMind trained their model on one of the largest corpora of Greek inscriptions to obtain results similar to those obtained by human epigraphers. Similar approaches have also been attempted with other languages such as the Scythian language, 3000-year-old Chinese oracle bone scripts, and Persian cuneiform tablets.

"Just because you can read the letters, that doesn’t mean you know what they mean" – Regina Barzilay (source)
Although it is not easy to decipher ancient inscriptions when the language is known (Greek, Latin, and so on), there are cases when the language is lost. Languages can be grouped into different families according to the traits they share (alphabets, vocabulary, grammar, sounds, and so on). Often languages share a common root ( e.g., Neo-Latin languages, "aquam" in Latin, "acqua" in Italian, "agua" in Spanish) and then undergo an evolutionary path that causes them to diverge. These principles have been used by linguists, to look for similarities and common patterns in order to decipher dead languages.
AI has been shown to be capable of finding patterns and finding similarities. Therefore, this approach has been attempted to try to decode lost languages such as Ugaritic or Linear B. In this case, the authors used a model based on LSTM and embedding and obtained some interesting results. The approach used, does nothing more than look for spans in the lost texts with known tokens.

Parting thoughts

Recent years have seen a large effort to digitize art. Major museums (such as the MET)have created large databases of paintings, books, statues, objects, artifacts, and so on. Often these huge digital libraries are accessible to the public and scholars free of charge. The institutions themselves are moving towards coordinated initiatives. For instance, the European Union has created guidelines to digitize the immense cultural heritage.
These initiatives are aimed at democratizing access to collections and cultural heritage. Indeed, many works preserved in museums can be seen by the public simply by visiting the sites. Moreover, museums only display a small part of their collections (many works are stored in warehouses and are never exhibited except in rare exhibitions) so these initiatives allow access to works that normally could not be seen. On the one hand, these initiatives allow valuable resources for scholars who want to study the works. On the other hand, algorithms need data and this allows increasingly sophisticated artificial intelligence models to be trained.
As we have seen, works, however, preserved in a museum, are delicate objects and can be lost at any time. Safeguarding them is an important and difficult job, whether it is to preserve them or restore them. In these tasks, artificial intelligence can help (monitoring, studying them, and so on). On the other hand, technology needs to be backed by policies and investments to preserve heritage.

Not least, AI has developed at great speed in recent years and interesting perspectives open up for the increasingly powerful models. In fact, as we have seen many of the models used are convolutional networks that have proven to be efficient for tasks involving images. However, all other areas of machine learning can be used as well. In fact, for example, unsupervised clustering can be used to group ceramic fragments.
In addition, the first models that were used to reconstruct inscriptions and translate dead languages were based on encoder-decoders and LSTMs. Pythia also used the same architecture, but already the later DeepMind Ithaca article was based on a transformer-like architecture. As vision transformers are shown to be effective for imaging we can expect more similar models in the coming years.
Another note is how it is not necessarily necessary to develop new and sophisticated algorithms for new tasks: as we have seen some researchers have leveraged DALL-E which is publicly accessible via a website. This shows that many of the available algorithms could be repurposed in the future.

Aside from technical demonstrations of AI capabilities, it opens up real perspectives and impact. For example, having a good digital image is the first step in restoring a painting or mosaic. Also, many archaeological sites are understaffed and surveillance is expensive and problematic. Not to mention that AI will be able to help make decisions, prioritize interventions, and reduce costs.
"These things look like cartoons. They don’t look like Klimt paintings. It’s like people who try to clone their dogs. You can do it, but it’s not the same dog." – Jane Kallir about AI reconstruction of lost Klimt paintings (source)
Open ethical questions certainly remain. The result as seen with the mosaics should be taken carefully. After all, the model sometimes more than reconstructing history seems to be rewriting it. As in the case of Rembrandt’s painting, the algorithm tries to guess based on the data at its disposal, and in fact, the curators of the exhibition pointed out the separation between the original part and what has been reconstructed.
In addition, satellite imagery and LIDAR technology have identified thousands of new archaeological sites not yet explored. These may be excavated by grave robbers long before archaeologists can work on them. As important as it is to consider publishing the data, one must also move cautiously.
Another thorny case is restorations. Since the 19th century, every new technology has been exploited in restoration without considering the possible damage. For example, concrete has burdened structures causing sometimes more damage than expected.
If you have found it interesting:
You can look for my other articles, you can also subscribe to get notified when I publish articles, and you can also connect or reach me on LinkedIn. Thanks for your support!
Here is the link to my GitHub repository, where I am planning to collect code and many resources related to machine learning, artificial intelligence, and more.
GitHub – SalvatoreRa/tutorial: Tutorials on machine learning, artificial intelligence, data science…
Or feel free to check out some of my other articles on Medium:
How artificial intelligence could save the Amazon rainforest
Reimagining The Little Prince with AI
Speaking the Language of Life: How AlphaFold2 and Co. Are Changing Biology