The world’s leading publication for data science, AI, and ML professionals.

Exploring chromatic storytelling in movies with R: the ChromaR package

Practical insights on how to generate movie framelines and many other chromatic data-viz plots

0. Where we left off

In the first part of this colorful journey, we explored some interesting chromatic examples picked up from famous movies, trying to guess the reasons behind directors’ choices and scratching the surface of what can be done through R scripting.

Exploring chromatic storytelling in movies with R (Part I)

In this second part, we will discuss more technical details introducing the ChromaR toolkit, an R package that allows us to render all the chromatic visualizations previously shown in part I (and many more).


1. Setting everything up

Installing ChromaR

The ChromaR package is hosted on GitHub and must be therefore installed through Devtools:

If you want to check the source code, here is the ChromaR Git repository:

detsutut/chroma

The data

As mentioned in the first part, retrieving the tensorial components of a video source is a bit tricky in R and must be therefore implemented in other ways. However, ChromaR exposes two ready-to-go databases you can use for practice purposes.

  R     G     B     lum   duration  RGB     title          frameId
1 95.43 84.02 76.22 86.58 7020      #5F544C Nausicaa o...  1
2 70.16 72.72 70.92 71.76 7470      #464846 Laputa Cas...  2
3 75.58 82.22 74.08 79.34 5185      #4B524A My Neighbo...  3
4 85.45 93.22 88.66 90.39 5595      #555D58 Porco Rosso    4
5 58.40 65.24 61.49 62.77 7995      #3A413D Princess M...  5
6 77.02 87.91 91.37 85.03 6180      #4D575B Kiki's Del...  6

If you already have your own csv RGB file, you can import it via getFrames().

Hayao Miyazaki and (some of) his colorful creations [© Reel Anarchy]
Hayao Miyazaki and (some of) his colorful creations [© Reel Anarchy]

2. Getting our first frameline

Sticking with the miyazaki dataset, let’s retrieve a frameline for Nausicaa, the first entry of our movie list. Framelines are generated by merging groups of consecutive frames (i.e. array of averaged RGB channels). The frames are grouped together using the groupFrames function and specifying the width of the time window. Depending on this parameter, the resulting frameline could be sharp or blurred. A 5-seconds window is usually fine for most of the movies, but you can visually tune it using the plotTimeWindows function.

The algorithm suggests two different cutoffs: a soft one (sharp frameline, most of the color information is preserved) and a hard one (blurred frameline, defined by the time constant of the exponential decay)
The algorithm suggests two different cutoffs: a soft one (sharp frameline, most of the color information is preserved) and a hard one (blurred frameline, defined by the time constant of the exponential decay)
...or you can simply pick up your favorite window size by visual inspection
…or you can simply pick up your favorite window size by visual inspection

Everything between 3 and 20 seconds seems pretty much ok, so let’s pick the first suggested window size. The plotFrameline function will take care of the rendering.

If you want to render multiple movies within the same plot, you can apply the groupframe function to each movie of the list and feed the whole collection to the plotFrameline function.

3. Summary Tiles

A chromatic study on entire seasons or movie franchises can be focused on many different features. Let’s consider now that little piece of gold which goes under the name of Love, Death & Robots, from Netflix.

"Three Robots", one of the most loved episodes of Love, Death & Robots by Blow Studio (© Netflix)
"Three Robots", one of the most loved episodes of Love, Death & Robots by Blow Studio (© Netflix)

Each episode of the critically acclaimed new Netflix series is designed, directed, and animated by different artists and studios. This heterogeneity is clearly reflected in the episodes’ chromatic choices, which can be summarized and compared using the plotSummaryTiles function. We could compare, for instance, the brightness of each episode in order to guess which one is the "darkest". In order to do so, we must get the summaries first. Summaries can be retrieved using the getSummary function. These summaries must be fed into the plotSummaryTiles function specifying which feature we’re interested in.

Average Brightness value for each episode
Average Brightness value for each episode

Results indicate "Helping Hand" as the darkest episode of the series. This shouldn’t surprise you, as the plot is entirely set in space. The silver medal goes to "Beyond the Aquila Rift", which doesn’t need any further explanation if you’ve already watched it. The brightest episodes (by far) are instead "Alternate Histories" and "When the Yogurt Took Over": world domination on the light side for once!

"Helping Hand" vs "When the Yogurt Took Over" (© Netflix)
"Helping Hand" vs "When the Yogurt Took Over" (© Netflix)

The same process can be repeated with saturation and hues, simply by switching the mode parameter from lum to h or s.

Average Hue and Saturation values for each episode
Average Hue and Saturation values for each episode

Unsurprisingly, "Zima Blue" seems to be the coldest episode. In the next chapter, we’ll dig deeper into it with a more specific set of functions.

This frame from "Zima Blue" basically encapsulates the color palette of the whole episode (© Netflix)
This frame from "Zima Blue" basically encapsulates the color palette of the whole episode (© Netflix)

4. Temperatures, Channels, and Circles

As a simple story with a big meaning, this brilliant episode exhibits a very tiny palette compared with the others, without losing a single bit of chromatic power. The palette of the whole episode can be retrieved simply by applying a wide time window (15 s) when grouping the frames.

"Zima Blue" palette and keyframes (© Netflix)
"Zima Blue" palette and keyframes (© Netflix)

As expected, blue tints dominate the scene here. We can clearly see this trend by inspecting hues through the temperature function, which measures the distance of the hue of each frame from pure red. Besides two sections, the whole episode is biased towards blue/cyan.

Temperature comparison between "Zima Blue" (top) and "When the Yogurt Took Over" (bottom)
Temperature comparison between "Zima Blue" (top) and "When the Yogurt Took Over" (bottom)

Similar conclusions may be drawn inspecting the hue channel trend through the plotChannel function, which returns the derivative trend in addition to the channel plot. Compared with other episodes, the derivative of "Zima Blue" is pretty much zero for a considerable part of the story, which means that no significant chromatic changes happen within the same scene (and that’s why we have such a small color palette).

Hue channel trend and its derivative
Hue channel trend and its derivative

It is indeed out of question that the blue/cyan tints are the main protagonists of "Zima Blue". What about the other episodes then? Is blue really overrepresented? Let’s compare it, for instance, with the fifth episode of the series, "Sucker of Souls", using the colorCircle function. This function generates a chord diagram, a graphical method of displaying the inter-relationships between data. Each frame is arranged radially around a circle, with the color transitions between frames drawn as arcs connecting the source with the endpoint.

Chord diagram of "Zima Blue" (left) and "Sucker of Souls" (right)
Chord diagram of "Zima Blue" (left) and "Sucker of Souls" (right)

Here we can clearly see how different the cyan/azure and orange percentages are between the two episodes. These two hues are diametrically opposed on the color wheel and their representation is therefore decisive to convey warmness or coldness. We can generate these fancy plots running the following script.


Conclusion

Exploring and analyzing the usage of Colors in movies is awesome and might lead to unexpected discoveries. This series of two articles has barely scratched the surface of a much more complex world which is also an amazing gym for every data scientist interested in data visualization: I have to admit it, I had so much fun working on this! Leave your thoughts in the comments and check Part 3 if you enjoyed this too!

Please note that the chromaR package is still a beta and many improvements must be performed in order to make it 100% operational. Therefore, please report any bug or issues you’re having!

Exploring chromatic storytelling in movies with R: color palette extraction

Tommaso Buonocore – Author – Towards Data Science | LinkedIn


Related Articles