An arsenal of tips and tricks to crack decision-making.
The Ultimate Mission of a Data Scientist: Support Decision-Making
The fundamental role of a Data Scientist is to support decision-making based on data. For that to be accomplished a data-driven culture has to be nurtured and that status quo challenged.
Whether you live on a concrete jungle or not, you are constantly making decisions. In many cases they might be so hardwired that you don’t even think about them. Your brain just push you through it without even reasoning. What we call “experience” can play an important role here: specifically, against an optimal outcome on decision-making; especially when the “experience” does not make room for considering the data.
The fundamental role of a Data Scientist is to support decision-making based on data. For that to be accomplished a data-driven culture has to be nurtured and that status quo challenged. The barrier to successfully transitioning from a “gut-feeling” type of culture (System 1) to a data-driven type of approach (System 2) is the people.
Experienced managers and executives tend to believe on their “intuition” to, quite often, make high-stakes decisions. That has to do a lot with many different types of biases that we’ll cover here, but not limited to. But, also with decision making based on emotions, which research has proven to not be recommended.
Just to clarify, by executives and managers, we really mean anybody that is in charge of making decisions, regardless of his/her title on the business card.
The Five Most Dangerous Biases
Bias can be defined as, “a particular tendency, trend, inclination, feeling, or opinion, especially one that is preconceived or unreasoned”. Out of the many biases that impact all of us negatively, overconfidence is the worst. Explicitly, when comes down to decision-making processing.
“Our gut makes us more vulnerable to cognitive biases such as overconfidence” — Max H. Bazerman
Decision-makers must acknowledge that they functioning under certain types of biases. Without this realization, it will be extremely difficult to break the wheel of making poor decisions. Managers tend to blame everything else when things go down hill, but take enormous credit when they succeed.
Here I’ll cover a little bit of the main types of biases that can impact decision-making.
- Overconfidence: Research says that, “overconfidence may be the mother of all decision-making biases” [2]. And that the “excessive faith that you know the truth is one form of overconfidence” [2]. If you are too certain that you are making the best decision, you better be doing that supported by data. Remember, “garbage in, garbage out”. Just because you might be using data, that does not mean you’ll succeed every time.
- Confirmation Bias: This is an effort to find a justification to a selective matter. Here we’ll look for anything that can support and confirm our beliefs and values. That is, we are consistently finding excuses to explain our failures without taking responsibilities for them.
- Unrealistically Positive Views of Self: Typically, this can be seen as a going hand in hand with overconfidence. In this bias, the person has an extremely positive view of him/herself. Specially, people that rely on experience to make decisions, tend to get a very strong positive view of themselves. A pill of humility every morning can be an effective remedy here.
- Illusion of Control: Usually, this bias comes attached to high levels of a superstitious mindset. One tend to believe that he/she can control random events by relying on things that they can do. Like, “if I wear a boot, it won’t rain”.
- Self-serving Attributions: I’ve already mentioned that before, but now let’s reframe that again: we tend to take a lot of credit when we succeed, and too little blame when we fail. Generally speaking, an executive will blame a failure to bad lick or other factors beyond the decision maker’s control, but credit their success to their skill or experience.
There are many others types of biases, but I wanted to give you a flavor of what we think to be the main ones when comes to decision-making on a professional setting. In essence, people are inclined to make decisions based on emotions rather than rationally, and this put us straight to a bifurcated roadway.
The War the Never Ends: System 1 and System 2
In a nutshell, System 1 relies on emotion, System 2 on rationality. As human beings, we all have both of these systems of thoughts.
System 1 is based on experience, and requires little cognitive efforts to be utilized. A classic example is the answer to 2 + 2 = ?. You see, you don’t even have to think much in order to find out the right answer. It’s an automatic and very fast processing.
Likewise, System 2 demands more cognitive effort to be utilized. An example of that is to find the answer to 16 x 23 = ?. For a vast majority of people, finding the correct answer will take more time and more effort. It’s not that automatic anymore.
Why is this relevant for decision-making anyways? It turns out that, when taking important decisions, we want to use System 2, and avoid the pitfalls of System 1. Because, System 1 constantly runs in our brains and never stops — it operates sub-consciously — if System 2 happens to be overloaded — with a lot of many important decisions to make — System 1 will take control and leading you to a decision that, quite often, are not optimal. When this happen we ended up making “good enough” decisions, and increasing the odds of failure.
Risk Averse, Risk Seeking, and Risk Neutral
The above chart shows the three different types of approaches when comes down to risk management. This is widely known in Economics and Finance.
Well, what this has to do with decision-making? Everything!
People make decisions not only based on monetary value (X axis), but also based on preferences (Y axis), like effort required and/or happiness. This unit of preference is called “utility”, and displays one’s preference for outcomes.
Utilities are important because they have influence on people’s decisions, and also help us to better understand their attitude towards to risk. In Economics, we see risk as the variance around a mean or expected value. On that regard, we can classify people in these tree categories: Risk Averse, Risk Seeking, and Risk Neutral.
Many of us are risk seeking for losses, and risk averse for gains. That is to say, we’ll take a sure thing, but avoid a sure loss. Because losses are more painful than the gains’ pleasure, we want to mitigate them. So, typically, the way executives see “risk” is to be the likelihood and the extent of loss, and not the variance. Hence, they most likely will focus on the downside of risk, and overlook the opportunities. How do you think this can impact decision-making? I’ll leave that to you.
Some Tools to Devise a Better Decision-Making Processing
Alright, you gotta have some tools on you belt to help you out, when you are summoned to a meeting you weren’t even aware of.
Let’s scratch the surface of three basic tools you should know about. Bear in mind that this is not an extensive list.
- Decision Table or Decision Matrix
Using a decision table or matrix allows us to come up with a total score that is based on the criteria we’ve selected as most important and the weights we have given to them, according to our preferences (remember the Utility Function?). Although, very basic, the decision table or matrix can be very useful when guiding the decision-maker.
2. Decision Trees
Decision Trees are quite useful to visually the path we are going and the complexity of our decisions as well as the risks and potential outcomes. Machine Learning Engineers and Data Scientist are familiar with decision tree, particularly, because the Decision Trees algorithm can be used in many solutions out there. Check the original article here to learn more.
3. Game Theory: Nash Equilibrium
Doing business is to be under the scrutiny of a non-cooperative game. From a business stand point, monopoly would be a dream come true, which means no competition. However, in many industries the competition is fierce. So when companies have to take decisions, the assumption is that they don’t know what their competitors will do. In essence, there is no cooperation when companies are fighting for market share. For that Nash Equilibrium is a great tool. More here.
Skills a Data Scientist Needs to Master
Nope! Your main goal as a DS is not to argue whether or not Python is better or worse than R. This is pointless, and a waste of time.
Rather, your ultimate mission is to support decision-making by fostering a data-driven culture that allows uncertainty to be mitigated, and gut-feeling decision making to become less regurgitated.
With that in mind, I’ll cover here three soft skills you must work on, if you want to break the wall of resistance when trying to change a culture.
- Communication
Business people don’t actually care which algorithm you used to come up with your analysis. Stop trying to look smart and to overcomplicate things. Emphasize the solutions you found to their problems, and not the tools you used.
By simplifying things, chances are that you’ll be heard and your message properly decoded. This is important to build trust and rapport because, in general, we like people that look like us.
As data advocates, we want to open the doors for a decision-making process that relies on data. For that, a great communication model is paramount.
2. Negotiation Modeling
Now, you’ve conveyed your points successfully. But, you find yourself running in circles and trying all you can to implement that, right?! This is where having a knowledge of negotiation can be handy.
However, don’t put too much effort on winning every battle though. Focus on the war. It’s okay to lose few battles if winning the war is the final goal. If you are just trying to make your point across and prove you’re the smarter person siting on the table, you might fail miserably in the long-run.
A great approach is to have a “win-win” attitude. That my friend, can really open the doors. If things got intense, you may want to prepare a good BATNA (Best Alternative To a Negotiated Agreement), which can help you to get what you need without damaging the relationship too much.
3. Managing Change
At the end of the day, all boils down to change. And this can be very challenging and it’s more likely to come from a top down decision. Meaning, some higher executive has a data-driven culture and want s to implement that across the firm.
In general, people tend to be resistant to change. Understanding the steps one goes when dealing with changes is crucial. Remember, creating a data-drive culture is a process and takes time. It’s really a marathon, not a sprint. Be resilient and keep trying to convert and influence others as you navigate towards to a democratization of decision based on data. Persistence is what you’ll need to practice to achieve the integration.
“Data that is loved tends to survive.” — Kurt Bollacker
Big Data and Internet of Things (IoT)
Why decision-makers need Data Science? Simply because, nowadays, the volume and structure of data (Big Data) we have available is immense, and to make sound decisions we need to have data mining capabilities to extract and analyze what matters.
Think about for a second, today, we have billions of devices connect to the internet 24/7. This generates a lot of data every second. This phenomenon is what we call “Internet of Things” (IoT). Which is directly and positively correlated to the amount to the volume of data we dump in the “big data” world.
In reality, it’s impossible for a human being successful when analyzing all of that without the machine. The recent advances in hardwire and software has opened windows of opportunities for Data Scientist to thrive and succeed when modeling solutions to companies’ problems.
Executives and general leadership cannot close their eyes on what’s happening right now. They must invest on building solid data structures and data warehousing so the analyst can do their job and bring competitive advantage to the table.
Making decisions based on when and/or where the wind blows is no longe acceptable.
Final Thoughts
All things considered, there are many other topics we didn’t have the chance to cover here. This article aimed to provide a high level of knowledge, and group some resources and information to help you when trying to understand the intricacies of the decision-making process.
Technical skills are necessary for the data scientist to perform his/her job, but business knowledge and soft skills are game changing when comes down to nurture changes in the company’s culture towards to a data-driven democratization.
Here we saw the main biases on the way of decision-making, how System 1 and 2 works, some tools to support decision-making, and some skills a data scientist needs to be successful when fighting status quo.
I hope this can motivate you to learn more. See you next time.
References:
[1] Richard Packard. Why Do I Keep Making Bad Decisions?! BHNW Buddha.
[2] Moore, D. A., Tenney, E. R., & Haran, U. (in press). Overprecision in judgment. In G. Wu and G. Keren (Eds.), Handbook of Judgment and Decision Making. New York: Wiley
[3] Harris, Christopher & Wu, Chen. (2014). Using tri-reference point theory to evaluate risk attitude and the effects of financial incentives in a gamified crowdsourcing task. Journal of Business Economics. 84. 281–302. 10.1007/s11573–014–0718–4.
