Why Is This Chart Bad? The Ultimate Guide to Data Visualization Evaluation using GoDVE (Grammar of Data Visualization Evaluation)
Don’t you ever look at graphics that go viral from time to time and try to analyze them? Well, I do. And most of them are crap. But why are they crap?What makes a bad visualization bad?
Today, I’ll tell you about VisQual — visualization quality tool and criteria used to evaluate data visualization.
DISCLAIMER: If you are here to answer the title questions skip ahead and go to chapter “Grammar of Data Visualization Evaluation”.
First, let’s start with a few example, because it’s the best the best way to learn! Here is a graph of cases of the global pandemic with an epi-demonic-al color palette.
What is wrong with this chart? Well, it’s the color palette which allows you to understand exactly and precisely nothing. Maybe not nothing per se, but it is really “misleading” at least. The difference between “None”, i.e. no cases and “10,001 or more” cases is huge (at least 10,001 to be precise). But the colors representation is identical. Is it? Not exactly. I checked it for you with GIMP.
Yeah, very different.
But let’s be fair and note the map has been changed. I am really refraining from using the word “corrected”, because the problem still persists — in a smaller scale but still.
How do you distinguish “5,001 to 10,000” from “1,001 to 5,000” or perhaps “10,001 to 20,000”?
Don’t even get me started on the whole colorblindness topic which is completely disregarded on a GOVERNMENT visualization.
BTW: 1 in 12 men and 1 in 200 women in the world are color blind
Enough with colors, let’s talk about scale and axes
Color is not the only popular mistake that is made in datavis. To be honest, the most popular mistakes regard the scale/axes shenanigans. To recall:
Scale refers to the sheer size (e.g. of bars in bar chart), while the axes describe the “scope” of the data space, helps you read the exact values and generally defines the coordinate system.
Speaking of the scale, let’s take a look at this Livestock vs Vehicles comparison. Last time I checked 300K < 4M, but who am I? Some kind of smarty-pants with a ruler?
“Next example regards axis. You know what is better than a climate change denier? A climate change denier who know 150% of data visualization.
Here is the the first bar chart from that tweet.
The problem of using Kelvin scale is that temperature rarely falls to the absolute zero (on Earth) and in general is between −68 °C (−90 °F or 205 K) and 56.7 °C (134.1 °F or 329.85 K). So the “absolute zero” scale maybe a little out of place here. I feel a bit disgusted having to explain that.
A few more examples for you to judge
We could go on and describe more and more terrible examples… and I know you are hungry for some hate-watching, but in order not to drag this part too much I’ll leave a few for your consideration as homework.
Guess what is wrong with those! If you are unsure check the related Reddit posts (and whole r/dataisugly community which I owe a huge shout-out to!)
This is the end of the rant part.
This is the data science part!
Now let’s talk about the real data visualization. We know where we are heading — toward VisQual, a data visualization evaluation tool and criteria
…but where did we come from?
What is a Data Visualization? (DO NOT SKIP this part)
A proper first step is to place the whole data visualization domain in some super domain. Some would say that it’s Data Science. I’d like to argue with that. Although, part of data visualization is certainly a subdomain of data science, a different part of it lays in… Art.
Yup, art as in Fine Arts and Applied Arts, which it both touches upon. It is a vague definition, but the whole definition of data visualization is also vague.
“The act of representing information as a picture, diagram or chart, or a picture that represents information in this way”, so simply anything. Anything that “visualizes data” or “represents information” is a data visualization. It’s art after all and anything can be art.
The more “fine” a graphic is the closer to the Van Gogh or Picasso-like art it gets and vice versa. The more “applied” or, “practical”, if you will, it is, the more data-scientish it becomes.
Although, we may argue, whether a particular graph is more beautiful or useful, we may evaluate it and that is the crucial part! The main assumption of VisQual is that we may judge a visualization and say that it is “good” or “bad”. However, I would hate to be purely subjective with it. Hence there are precise criteria, which we will get to eventually.
Let’s talk about grammars first. Wait, “grammars”? Yes! Grammars of graphics, the origins of datavis (de)composition.
We will mention two big organizers and their ideas derived one from another.
Grammar of Graphics
Don’t worry, we will not talk about formal grammars, Turing machines and automata theory. The concept introduced by Leland Wilkinson in The Grammar of Graphics sorts a graphic (e.g. a chart) into a context-free grammar that consists of elements such as:
- Data
Let’s say we have date of birth and deaths of some people. - Variables
…are their ages (date of death - date of birth) - Algebra
…defines how to subtract the numbers (here subtracting dates). - Scales
…defines how big a unit of our bar chart is. - Statistics
If we want to create a chart of average lifespan, then go for it! - Geometry
…tells the graphic whether it should be a bar chart, scatter plot etc. - Coordinates
Probably we are working in Cartesian coordinates, but heck, sometime it’s polar! - Aesthetics
…defines how far the bars are from each other, where the axes is placed (left, right, top or bottom) and other aesthetic aspects. - Facets
Perhaps we want to split our chart into many charts based on the nationality of the people? - Guides
…is basically all “text” that the visualization contains (title, axis title, legend etc.)
Wilkinson’s idea was brilliant in its formal and engineer approach and it is a great foundation to built on top of.
A Layered Grammar of Graphics
Wilkinson’s concept was undertaken by Hadley Wickham in his A Layered Grammar of Graphics where he changes the approach of formal description into practical application. Creating the first iteration of ggplot, he introduces his concept which splits a graphic into layers:
- Data
- Mapping
- Geom
- Stat
- Position
- Scale
- Coord
- Facet
If you are wondering if there is any relation or, ahem, mapping, between GoG and Layered GoG — there is! That is how Wickham defined it.
We could talk about how awesome GoG and Layered GoG are, but…
But we are not here to create or disassemble graphics. This is what Wilkinson and Wickham did. We are here to evaluate them. Grammar of Graphics or layered grammar of graphics were created to design/build/decompose a graphic. We are here to assess. So put away your elements and layers and let’s talk about categories of…
Grammar of Data Visualization Evaluation
GoDVE (still working on the shorthand) is the baseline for talking about data visualization evaluation. It consists of 11 categories with 118 criteria in the form of simple yes/no questions. To evaluate a visualization you just answer the questions. If the answer is “yes” then all’s good, but a “no” means that the visualization has an issue. What kind of issue? Let me explain with an example — the best way to learn!
Let’s take a look at a non-ideal visualization from r/dataisbeautiful and mention some questions that it would trigger. “Some”, because ain’t nobody got time for all. It’s ok to evaluate a datvis only in some aspects. If you want to go through all of them, go on! (Check out the tool introduced in the last chapter!)
I’ll try to tackle each category to give you a good impression of what they are about. Let’s dive in!
Data
Is the data source provided? YES
Although most of the trendy, viral visualization do not bother to, this one actually provides the source (“inspiration”) in the bottom right corner. This is already a very, very good sign.
Aesthetics
Is the information organized according to five hat racks? YES
The data is very well organized chronologically.
Does the visualization obey the reading gravity? NO
Well, tbh, time usually flies from left to right. Here it goes “towards” the viewer, which some (including me) find not very intuitive.
Are all axes’ directions and shape proper? NO
The time axis is weirdly and needlessly deformed. Distance from 200 to 300 is a lot shorter than from 1900 to 2000. A century is a century.
Coordinates
Is the scale complete (not cut)? YES
I am bringing this question to highlight that it is one of the most common mistakes/manipulation in datavis. Fortunately, the axis is “complete”, i.e. no years/decades are “cut off” like in the abomination below:
Is there enough axes or value references? NO
There is no axis for the percent of population who died due to each pandemic.
Are all used dimensions necessary? NO
Being aware of the… “eye-pleasing effect of the hairy balls”, which we will get back to later, the third dimension could be avoided and reduce the volume illegibility and confusion.
Position
Is the alignment proper? YES
Although my OCD kicks in, when I look at the unjustifiable justified to the left main panel, the alignment of most of the elements is… correct.
Does the visualization utilize the space properly? NO
There is a lot of empty space on the right.
Guides
Is there not too many labels? NO
There is too many guides in too many places. Moreover, some of the super crucial labels (like the note about volume percentage in the top left corner) are not highlighted enough.
Geometry
This is certainly the yuckiest yuck of yucks of this visualization.
Here we go…
Is the shape-to-data conversion easy to interpret? NO
Volumes are not easy to interpret and compare for humans. We think linearly. We compare lengths easily, areas with difficulty, volumes… we can’t compare volumes reliably.
Do the shapes allow easy comparison of the data? NO
Look above.
Are the additional dimensions of shapes showing additional information? NO
Volume presents a piece of information that requires only one dimension. Hence, it could be reduced and is certainly not necessary.
Do the shapes not prevent reading the data? NO
One thing is that there is the 3rd dimension and the whole cant-read-that-volume thing. Second thin is the perspective. Do you think that these balls are similar in size? I have no idea, they look similar, but…
Let’s check!
Antonine Plague took place in 165–180 AD. The world population in these times was nearly 190 000 000. During the plague 5 000 000 people died. It makes for ~2,63% of world population. How about COVID?
We take 35 000 000 (state according to the visualization) divide by 7 794 798 739 and get… ~0,45%, which is nearly 6 times less. But balls look nearly the same. Is it perspective? Is it the cant-read-that-volume thing? I cannot even.
You could say that the authors did their best to make the data look as fancy as possible while making it as unreadable as possible. Let’s move on.
Do the shapes represent the data in proper scale? NO
Do we even know what is the data? Volumes represents percent of population killed by diseases. We are also given the absolute value. Now I would have to check the world population in given times, divide, and… probably realize that a datavis that makes me check the data that it is missing is not fulfilling its purpose.
Do the shapes represent the data precisely (without approximation)? NO
Oh boy. Can anyone here estimate the volume of the hairy balls with flying fibers? (BTW, I you want to quote this article, quote this sentence).
Can anyone here estimate the volume of the hairy balls with flying fibers?
Statistics and Facets
No questions in those categories are appropriate/interesting for the visualization, so we skip it.
Theme
Is it protanopia-friendly? YES
Is it deuteranopia-friendly? YES
Is it tritanopia-friendly? YES
The colors do not represent any data, so… I just wanted to make a shout-out to all people how unfortunately can’t experience the world in all colors.
Manipulation
Is the data displayed in a unbiased fashion? YES
Is the data not manipulated? YES
It’s difficult to say with 100% accuracy what are the intentions of a person. Hence this category is generally very “I think/Perhaps/Probably …”.
I think that it is very difficult to make any argument that the author had any intention (or potential gain) in showing the data in a biased/manipulated way.
Does it contain no items with illusory effect? NO
The balls again. A perfect way to make an impression that things are x times bigger (length/height) while their are in fact x³ times bigger (volume).
Subjective
I’m skipping this category all together, because you probably got what is my impression of this chart. But check how you would answer the questions!
- Is this visualization interesting?
- Is this visualization better than similar ones that you have seen?
- Do you like the theme of the visualization?
- Is the visualization aesthetically pleasing?
- Do you understand the visualization?
- Do you like it?
- Did you learn something from this visualization?
- Is the presented information useful?
- Was the time spent experiencing the visualization worth it?
- Would you like to see a visualization similar to this one?
Final word(s)
Whenever you think that you made a terrible chart, remember that there may be over 30,000 people (with upvotes) who beg to differ.
The above visualization is terrible to me, but it is not terrible (nor excellent) in general. That is the most important clue about VisQual: it helps you evaluate the visualization and it indicates precisely the mistakes. However, it does not give a mark or a score. You don’t get 94/100 or 3/10. Even though you could calculate a final score with simple fraction of number of positive answers to total number of questions and compare it with others, you MUST NOT do that.
VisQual is not a theoretical marking tool, it is a practical feedback tool.
Speaking of practical tools…
Checkout full VisQual and evaluate your chart online!
Want to dive deeper? Checkout all categories and questions and evaluate your own visualization using VisQual online!
Thanks to @michbur for his advisory.
