Gut feelings and science haven’t always gotten along. In fact, anything that serves to simplify the process or outputs of science with imagination or "feelings" have historically been met with strong opposition. As Leary put it when discussing the use of feelings and imagination in science:
"Science has tried to legislate this ambivalence out of existence by dictating the acceptable forms of sensibility and by limiting the free play of the imagination."
Data science is no different. Indeed, others have argued in a similar vein that data science may not be a good profession for gut-inspired decision making. And this argument is made with the utmost nobility in its intended ends. Intuition, common sense, and guts have failed us in the past because they are not only imbued by experience but, consequently, also intertwined and knotted with normal, natural human bias.

So, let’s make a distinction. Relying on intuition and only intuition is very different than allowing one’s decision making to be informed by intuition. That is, intuition is information that, although biased, is still enriched by our unique and often difficult to describe experiences and so have value in our decision making process. In matters of science, data or otherwise, intuition is still valuable but should not serve as the sole source of any final decision.
To bring this point home, research on intuition has shown that in highly complex industries and/or tasks, expert intuition can bring about quicker more accurate decision making.
Data Science is a complex industry, that is also characterized by the combination of many complex tasks. Moreover, because most data scientists are employed by businesses and not research institutions, the need to make decisions quickly is a requirement for building successful solutions. Therefore, I argue that developing expert intuition and learning how to use it to inform decision making can help data scientists become better data scientists.
So how do we develop expert intuition?
Expert Intuition is defined as a…
"… skill that develops in a domain after an individual with innate talent has considerable learning experience accompanied by an awareness of the quality of each performance."
To develop such a skill requires experience, but not just any experience. It requires what I refer to as "tinkering" experience. Take statistical modeling as an example. Understanding how statistics work is a core requirement to all data scientists, yet when we look at the gamut of models available in the data science toolkit and the various applications of mathematics in those models, it is futile to expect that any one data scientist know all their mathematical derivations.
What we can come to expect is that data scientists have an intuition for how those models work because they have experience tinkering with inputs and hyperparameters and watching them generate different outputs. It is exactly this type of tinkering that generates the type of expert intuition required to make quick decisions in model selection and development that enable data scientists to go from an impossible world of possibilities to a limited set of "most likely" inputs and hyperparameter values.
Expert intuition in these instances can also help us look for more creative combinations and opportunities in our development efforts. Having a general intuition for the behavior of unsupervised algorithms, for example, may help to inform a more creative approach to a modeling problem because the developer knows how to reduce the dimensionality of the data without losing too much information from the feature set.

To enable the development of intuition in data science, we must ensure that we can tinker and to tinker effectively we need…
To Imitate
Don’t be afraid to imitate. All learning starts with imitating and so in order to become good intuitive data scientists, we need to learn to be good imitators. But not just any sort of imitation will suffice. We need to imitate the best, the best solutions, the best code, the best deployments for different situations. So, while you are imitating (e.g. following tutorials, working through blog posts, going through books or courses), don’t let yourself get stuck if things aren’t working.
Community
Find a community that makes you feel safe while you experiment with your expanding skill set. Make sure it is a place where you can express your intuition without feeling judged.
Inspiration
Remember that learning is not the filling of a bucket but "the lighting of a fire" – Yeats. Whether or not Yeats ever said these words when talking about education is less important than the value this perspective provides. Learning is inspiring and so, we must seek inspiration in our pursuit to expert intuition. It is an important mindset to have because buckets are fixed but fires are limited only by fuel. Learning is not fixed either and so our capacity to learn is more like a fire than a bucket of knowledge we may fill to the brim.
Creativity
As our experience builds, so too does our ability to be creative. Don’t forget the experiences you had before you decided to learn data science because those experience may serve as the basis for a novel and creative application of what you are doing today.
Reward
In data science, not getting errors in our code is rewarding, so too is seeing our model accurately identify, classify, or predict the information we intended to output. Take the time to be rewarded from these wins as allowing yourself those moments helps to tune and train your expert intuition as you pursue new solutions in the future.
Play
Finally, we need to allow time for us to play. Play with data, play with code, play with models, play with technologies…and the list goes on. What is play and how is it different from learning? Play is the space we develop in where there are no expectations or deadlines. We experiment with the intuition we have built. We allow ourselves to try things, even if we don’t know if they make explicit sense. We allow ourselves to make mistakes and to see those mistakes as further informing our expert intuition.
Thanks for reading!
Like engaging to learn more about data science? Join me.