The world’s leading publication for data science, AI, and ML professionals.

Sonification – When You Go Beyond the Visual Representation Of Data

Communicate and find patterns with sound as your guide

Does this photo look like something else? As sound waves perhaps? Photo by Adrien Olichon from Unsplash

"Without sound, celebration and grief look nearly the same." (Ben Marcus)

Sound offers something unique to the perception of our surroundings.

I always remember going into all these historical landmarks and seeing them completely furnished and pretty but at the same time inert and lifeless.

However, once you played contemporary music (or sound effects) in those chambers, they would suddenly become alive. You’d be immediately transported to that point in time.

You might be attending a grand ball in honor of Louis XIV or witnessing the writing of the Declaration of Independence.

Sound brings life to objects, places, and moments that lay dormant otherwise. The same happens when you sonify data.

And it’s the next frontier of sound development.


What’s data sonification?

It’s another way of conveying information.

When you present data, you usually use visual cues to show the analysis of your work. But in this case, you use sound to make something easier to understand.

Think of a Geiger counter. This is an electronic device that measures the level of radiation in its surroundings. If you went to Chernobyl, you might take one of these and avoid getting too close to places that can damage your health (permanently).

To keep you safe, this device will emit audio clicks that represent the level of radiation you’re exposed to. The more clicks you hear (with fewer spaces between them), the higher the radiation.

That would be your "get the hell out of there" signal.

Here’s a more visual representation of data translated into sound. In this case, the composer and computer scientist Brian Foo wanted to represent the changes in Bejing’s air pollution over a 3-year timeframe with both sound and visuals.

On top of a base layer of music (a repetitive piano-like sound with a moderate tempo), Brian created an algorithm to alter the sonic and optical elements of this song based on the city’s daily air quality measurements.

Screenshot 1 - Sonification of Bejing's air quality via Wikipedia
Screenshot 1 – Sonification of Bejing’s air quality via Wikipedia

On the one hand, when air quality deteriorated you would hear more percussive and noisy elements in the music [screenshot 1]. This would also be accompanied by fast-paced notes and a higher density of dots in the graph.

On the other hand, when air pollution improved [screenshot 2], these rhythmic sections would fade away and the music would become more pleasant, which also resulted in a less saturated graph.

Screenshot 2 - Sonification of Bejing's air quality via Wikipedia
Screenshot 2 – Sonification of Bejing’s air quality via Wikipedia

With this sonification experiment, Brian created a unique perspective on a pressing matter and helped raise people’s awareness of the issue.

These are just two examples of how sound feedback can help you perceive the world in a different way, and sometimes more easily. Sound can be both a useful and a powerful tool to convey data sequences.


What techniques do you use to turn data into sound?

The same data from which you can create visual charts, you can also turn into sound and music. You could use different sound elements (pitch, volume, tempo) to make the user aware of changes in the information you’re trying to get across.

For instance, Ambrose Soehn co-founder of an Audio branding firm called ChirpShop used changes in pitch to raise awareness about climate change.

By taking historical data on global temperature changes, he assigned higher pitches to more extreme climate events (e.g. heat waves), and lower pitches to cooler years.

In other words, for each year between 1880 and 2016, he would assign a specific pitch (in MIDI format) to a temperature value.

Once he had them mapped out, he’d import that file into a DAW (Digital Audio Workstation) where he would assign a virtual instrument to that sequence of pitches (notes really) to turn them into music.

And voilá! Here’s the resulting sound graph of this experiment.

What’s the upside of doing this? Data becomes alive! What was once just a conceptual grasp of a graph, now becomes an emotional ride to a deeper understanding of the information. You get involved with what you hear and are able to feel the data.

It’s not just a stream of numbers anymore, it’s a personal connection to the wisdom behind them.

There’re also other ways of hearing information:

  • Audification: it’s a way of representing data values into sounds within our range of hearing (20Hz to 20kHz). For instance, when your doctor places a stethoscope against your skin, they’re able to hear sounds made by the heart, lungs, intestines, and even blood flow through your veins and arteries. They’re turning the organ’s sounds into something they can easily hear.
  • Model-based sonification: it’s a system of acoustic responses that are generated when someone interacts with it. You establish a set of instructions (e.g. when the user approaches, increase the pitch), and tell the user how to interact with it. For example, if you’ve ever seen an instrument called theremin, you would know that depending on how you wave your hands around it you’ll get to play different pitches, articulations, and volumes, resulting in a quite particular timbre.

I bet you haven’t thought of sound like this before, have you?


A taste of audio representations

We perceive sound differently from images. While people say a picture is worth a thousand words, a sound will make you speechless. It goes beyond the realm of words and many times you’ll find yourself unable to describe what you hear.

But that’s until you found a frame of reference that will guide you to valuable information.

Let’s look at some examples to better understand this.

The most basic one is the ticking of a clock. Without the reference to time division, that sound wouldn’t convey any useful data. But once you understand that it ticks every second and rings every hour, you’re able to use this information more appropriately. Basically, what you hear is an auditory cue (tick/ring) to represent a piece of data (second/hour).

Then you have different degrees of complexity.

You can use sound to represent images.

Imagine you have a photo and you want to transform it into sound. A software called Photosounder allows you to turn pixels into low and high pitches. You process an image on this software from left to right.

Every pixel contained in the photo’s main figure will get a distinctive pitch. So if it’s a circle you would first hear a pitch in the middle, followed by two pitches going in opposite directions (one higher, one lower) and ending with a middle pitch.

Here’s an example with a banana to make it easier for you.

An even more complex set of data is translating the amino acid sequence of the Covid-19 spike protein into sound. Thanks to the work of Markus Buehler, professor of engineering at MIT and music composer, researchers can now get to hear its structure displayed through the lens of sound.

This could stimulate scientists to look at the problem from another perspective and enhance lateral thinking. If you’re curious, here’s the actual sound representation of the virus.


The takeaway

One of the tricks we use to understand data is by displaying it as charts, graphs, infographics, and so on. In addition to showing data in visual terms, why not try to represent it as sound?

There’s a whole area of development called Sonification that can aid people not only to communicate ideas to others but also help us perceive patterns and trends that might go unnoticed through traditional visualizations.

And it helps bring information alive. You feel differently when you’re able to hear things rather than just see them. The space around you fills with vibrations that can reach the deepest parts of your soul. It feels meaningful.

With sonification, you get to add an emotional understanding to science.

And when you get to experience it, Sound and music become powerful tools but only if we learn how to use them appropriately.


If you enjoyed the article try reading these:

How to Become a Music Sommelier

When Fans Become Investors – The Use of NFTs in the Music Industry

We’re Entering Into a Post Screen Future – Sound Is the Way


If you enjoy reading stories like these and would like to support writers on Medium, consider signing up to become a Medium member. It’s just $5 a month and you’ll have unlimited access to articles from amazing writers all over the world.


Related Articles