The world’s leading publication for data science, AI, and ML professionals.

Going Beyond with Agile Data Science Workflows

Seeing beyond what my eyes are seeing is something I apply to every aspect in my life. In this article I'll show how Agile Data Science…

We live in a complex world, we live a cultural world. That means that we can’t choose being culture-less creatures. We are born and that’s it, you are already there. You can’t choose your status, parents, city and more. Then we learn, from life and from school, our friends, our family.

That process creates a way of seeing things, a way of thinking about the world, about ourselves and what surrounds us. Our senses open us to the world, we hear, we feel, we taste, we see. When I mention "seeing beyond" or something like that I mean the combination of all of our senses, what we perceive from every angle.

We get the data from the "outside world" and our body and brain analyze the raw data we got, and then we "interpret" things.

Modeling the world

What is this "interpretation"? Just what we’ve learned about how to react, think, feel and understand from the information we are getting. When we are understanding we are decoding the parts that forms this complex thing, and transforming the raw data we got in the beginning into something useful and simple.

We do this by modeling. This is the process of understanding the "reality", the world around us, but creating a higher level prototype that will describe the things we are seeing, hearing and feeling, but it’s a representative thing, not the "actual" or "real" thing.

Going further

We don’t stay there. Our model of the world, or a process, has another part. What we think that particular thing means and how we feel about it.

When I was younger I had a lot of prejudice about lots and lots of things. Judging things and people before even meeting them. And I don’t think I was alone in this.

We are used to jump to conclusions really fast, not analyzing every side of things. We are use to see what our eyes are seeing and "trusting our gut".

Sadly the common sense that reigns in our culture is Aristotelian and Medieval (Études d’histoire de la pensée scientifique – Alexander Koyré). That means that intuition fails a lot of times when trying to understand the world, also this "common sense" comes sometimes with judgement, something that creates a bias in the way we see things.

Going and seeing beyond in this context means going a step forward, putting your judgment, common sense and intuition aside and really analyzing a situation. We should be doing this for everything that happens around us, question ourselves if the thing you are doing, thinking and perceiving is actually correct. This is something very close to the Cartesian Doubt.

Why going beyond with Data Science?

Agility is fundamental to business’ ability to successfully build systems in a world where it’s difficult to predict the future – James Kobielus.

So what does Data Science have to do with any of this? Actually going further, and beyond our common sense and intuition is the only way of solving complex business problems.

In a world full of intuitive models, disruption and advancement come from going beyond, using data to understand what can not be seen with the naked eye or with an "expert look".

The process of an Agile Data Science Workflow proposed by Russell Jurney is an amazing way of understanding how and why Data Science together with agility helps us going beyond, seeing more and solving problems in a creative way.

https://www.oreilly.com/ideas/a-manifesto-for-agile-data-science
https://www.oreilly.com/ideas/a-manifesto-for-agile-data-science

The manifesto for Agile Data Science (we should put Agile Data Science workflow here) leads us to this. Iterating, over and over again, rethinking the business process and needs, experimenting a lot, listening what the data has to say, understanding and encouraging the business to understand that the data’s opinion must always be included in product discussions, finding a critical path to solve the problem and then organizing the team around completing it, and going further, letting the models solve the problems, of course using our expertise to help them, but not biasing them.

From understanding the business and its needs to the deployment of the solution we need to take a look at the bigger picture, from above, from below, from the side. Emptying our minds from the intuition that everyone can add to the solution, and trust what our models are saying about the process and understanding how it is solving the problem.


As a final note I would like to remember and state that every model assumes things, and here model can mean a way of understanding how the world works to a Random Forest classifier to see if a transaction will be fraudulent or not. We need to understand this assumptions, they may not be clear in the beginning but they are there. In Machine Learning and Deep Learning is easier to see what are these assumptions, they are in the papers that proposed the algorithm and also in the codes, so before using a ML or DL library to import a model, understand what it assumes from the data and the process, that will make the "debug" so much easier.


Thanks for reading this. I hope you found something interesting here 🙂

If you have questions just add me in twitter

Favio Vázquez (@FavioVaz) | Twitter

and LinkedIn.

Favio Vázquez – Principal Data Scientist – OXXO | LinkedIn

See you there 🙂


Related Articles