The Making of Great Hypothesis

How Bill and Melinda Gates’ favorite book Factfulness will leapfrog your data science practice

Motivation

So far, data science education focus heavily on data wrangling, hypothesis testing and causal inference. Very rarely do we hear about the preceding step — hypothesis formation.

Forming the wrong hypothesis is costly

Once you’re set on testing a hypothesis, you can spend hours, day, months or even years gathering data, cleaning it, visualizing it, constructing fancy predictive and causal models to no avail. Your pet hypothesis simply did not work— nothing is significant!

That marks the beginning of your identity crisis. You start feeling sorry for the time you’ve wasted. You begin to question your smartness for the very first time in your life. “Maybe I’m not as smart as my 99th percentile SAT/GRE/LSAT indicates,” you dreaded. But you’ve invested so much. Can you really admit defeat so easily? Your fingers just can’t resist the urge to tweak the design ever so slightly, and see if version 1000 would be the magical design that conform to your theory. You think you’re an honest, upstanding citizen. But in your persistent pursuit of significance, you’ve just landed yourself in the land of p-hacking.

It happens to the best of us

The question is: how can we form better hypothesis? Last week, I bumped into just the right book: Factfulness. It talks about “10 reasons we’re wrong about the world, and why things are better than you think”. I have personally committed quite a few mistakes on the list. The key to forming smarter hypothesis, I think, lies in developing a deep understanding of our systemic biases. By side-stepping these tempting yet misguided hypothesis, we can save ourselves a lot of time and anguish.

Disclaimer

This blogpost is a cheatsheet, created for self use and sharing with friends. If you find it useful, I strongly encourage you to purchase the book on Amazon. I simply cannot do justice to the greatest paperback I’ve read in years. I’m sure you would be delighted by the masterful data-based story-telling. You might even find a lot of insights that are not in this cheatsheet.

1. The Gap Instinct

Definition

The all-too-human urge to see the world via a binary lens.

Example

Bottom x% vs top y%

The richest 10% in Brazil earns 41% of the total income. Disturbing, right? It sounds too high. We quickly imagine an elite stealing resources from all the rest. The media support that impression with images of the very richest — often not the richest 10 percent but probably the richest 0.1 percent, the ultra-rich — and their boats, horses, and huge mansions.

Yes, the number is disturbingly high. At the same time, it hasn’t been this low for many years. Statistics are often used in dramatic ways for political purposes, but it’s important that they also help us navigate reality…In reality, even in one of the world’s most unequal countries, there is no gap. Most people are in the middle.

Controlling the gap instinct

• Recognize a gap when you hear it.
• Averages can be deceptive. Try plotting distributions (over time).
• Most academic focus on average treatment effects, and use heterogenous effects as a robustness check. In real business settings, you should probably put heterogenous effects front and center.

2. The Negativity Instinct

Definiton

“The mega misconception that the world is getting worse.”

Example

• When surveyed, people think extreme poverty has worsened, and crime rate has increased. Using dozens of outcome variables from World Bank and UN, the authors show that it is simply not true. (I’m not posting these plots, because I might be infringing copyrights. But I highly recommend checking out for yourself.) For example, 2017 Zambia is better than 1891 Sweden.

Temptation

• “Most things used to be worse, not better. But it is extremely easy for humans to forget how things really did ‘used to be.’”
• Selective reporting

How to reign in our negativity instinct

• Recognize selective reporting — as stories about gradual improvement don’t sell.
• More news != more suffering. “More bad news is sometimes due to better surveillance of suffering, not a worsening world.”
• Recognize that most people romanticize the past.
• Don’t be obsessed with level, and be curious about the direction of change. Plot some time series data and see whether it was truly better in the past.

3. The Straight Line Instinct

Definitions

• Over-reliance on linear model
• Over-reliance on linear extrapolation

Example

“The mega misconception that ‘the world is population is just increasing and increasing.’”

Temptation

• The authors gave many examples of how people around the world, across different income and professions, love to assume straight lines.
• Personally, a related reason is that linear models have received a lot of academic attention. So if you use linear models, you can call on many fancy proofs and cool techniques to defend your strategy; whereas non-parametric still remain relatively under-studied.

Antidote

• Authors: Note that data can be distributed in a number of shapes: straight, S-bends, A-slide, humps, exponential, etc. They also talked about how different social and economic forces give rise to different shape. They are quite interesting. Go see for yourself :)
• Me: have a deeper understanding of the underlying forces that generate these data. Structural models, while complex and unwieldy, can be extremely useful at times.

4. The Fear Instinct

Definition

Fear can be useful, but only if it is directed at the right things. The fear instinct is a terrible guide for understanding the world. It makes us give our attention to the unlikely dangers that we are most afraid of, and neglect what is actually most risky.

Example

The authors plotted many time series on natural disaster, plane and car crashes, war and conflicts, etc.

Temptation

• Mistake media report frequency for real frequencies.
• “Risk = danger *exposure”. Most people forget about the exposure part, and worry unnecessarily about events they have low exposure to.

Antidote

• f(media report)!=f(event)
• “risk = danger * exposure”

5. The Size Instinct

Definition

We tend to blow things out of proportion, especially when:

• it’s related to us
• a statistic is singled out

Consequence

The two aspects of the size instinct, together with the negativity instinct, make us systemically underestimate the progress that has been made in the world.

Example

In the test questions about global proportions, people consistently say about 20% of people are having their basic needs met. The correct answer in most cases is close to 80 percent, or even 90 percent.At the same time, we systematically overestimate other proportions. The proportion of immigrants in our countries. The proportion of people opposed to homosexuality. In each of these cases, at least in the United States and Europe, our interpretations are more dramatic than the reality.

Antidote

Put the statistics in context: rather than reporting the absolute number, try reporting percentages.

6. The Generalization Instinct

Definition

Some generalization is needed. “Categories are absolutely necessary for us to function. They give structure to our thoughts. Imagine if we saw every item and every scenario as truly unique — we would not even have a language to describe the world around us.” But it becomes a problem when we over-generalize.

Negative Consequences

When you generalize result from one group to another wrongly, you risk

• Thinking there’s a business opportunity when there isn’t;
• Thinking there’s no business opportunity when there is.

Example

Every pregnancy results in roughly two years of lost menstruation. If you’re a manufacturer of mensural pads, this is bad for business. So you ought to know about, and be so happy about, the drop in babies per woman across the world. You ought to know and be happy too about the growth in the number of educated women working away from home. Because these developments have created an exploding market for your products over the last few decades among billions of menstruating women now living on Levels 2 and 3.

Antidote

Assume you’re not normal, and other people are not idiots

• Look for better ways to categorize people into groups.
• Across group: look for similarities, but don’t generalize result from one group to another, unless you have sufficient evidence to support it
• Within group: look for differences
• Be aware of anecdotal evidence: they are usually the outliers
• Be aware of English reporting with no data. For example, majority can be 50.00001%

7. The Destiny Instinct

Definition

Claims that X (people, countries, religions and cultures) are destined to have a certain outcome, because these are immutable characteristics.

Temptation

• In reality, countries, religions and cultures are shifting slowly but surely. Sometimes, people mistake slow change with no change.
• Doesn’t help that many best-selling books are perpetuating this myth.

Negative Consequence

Stereotypes, discriminations, and missed opportunities.

Antidote

• Slow change != No change
• Ditch the cultural/religion destiny discourse already.

8. The Single Perspective Instinct

Definition

When we’re seeking for the one explanation that will explain everything.

Temptation

People, especially scholars, love simple, elegant solutions.

Antidote

Get a toolbox, not a hammer

• Be your own devil’s advocate. Try to collect counter-examples to your pet narratives.
• Realize that the reality is complex. The corollary is that beware of simple ideas and simple solutions.
• Never trust a single source — even if that source is a subject expert. Realize that expert are only expert on the topics they study. The authors show that they are just as wrong on other subjects when surveyed.
• Watch out for the ideologue. “Even democracy is not the single solution.”

9. The Blame Instinct

Definition

“The instinct to find a clear, simple reason for why something bad has happened.”

Example

There’s no shortage of examples :) So I’ll skip.

Negative Consequence

• Exaggerates the importance of any individual/group.
• Detracts from understanding the issue, and coming up with constructive solutions.

Antidote

Resist the urge to blame everyone. Because the problem is that when we identify the bad guy, we are done thinking. If you really want to change the world, you have to understand how it actually works and forget about punching anyone in the face.

• “Look for causes, not villains.”
• “Look for systems, not heroes.”

10. The Urgency Instinct

Definition

“Now or never! Tomorrow may be too late! Special Offer! Today Only!”

Temptation

It has shown to be an effective click-bait and growth hack. It taps into our most primal fear instinct.

Negative Consequence

You’re making suboptimal choice: spending less time and energy on truly important tasks.

Antidote

• “Insist on data” that has relevant and accurate measure of urgency.
• “Beware of fortune-tellers, because any predictions about the future is inherently uncertain.”
• “Be wary of drastic action,” because the side-effects might be hard to undo. Better to make small, incremental improvements, and evaluate each remedy every step of the way.

The End

You’ve reached the end of this cheatsheet. But in a sense, this is just the beginning.

I’ve already caught myself falling into several of these traps in my own research. How I wish this book was written when I first started my PhD in Economics. It would have saved years of my life, literally. Thankfully, it’s never too late to recognize your own mistake, and learn from it :) Armed with these insights, I would be able to side-step tempting but untrue hypothesis.

Which one of these pitfalls have you seen in practices? Leave a comment below so we can watch out for one another.

Till next time!