Fixing Fundamentals Is Essential to Use Fancy AI
Three techniques can help companies build fundamentals that create foundations for data-based competitive advantage
From Business Intelligence to Big Data to Machine Learning to AI, the world of data has made blinding progress in the last two decades. Yet, ever so often researchers keep pointing to sobering numbers on the failure of companies to leverage data. Oftentimes lack of executive support, data teams not focussing on real business problems, company not having the right skill sets, etc are sighted as culprits.
In my view, oftentimes it is the lack of focus and patience to build the fundamentals. And plainly said, fundamentals are boring, tedious, and time-consuming to build. Who truly likes things such as governance and documentation? My guess is neither the data scientists wanting to experiment with new technologies nor the executives looking for quick insights and outcomes in the world of quarterly reporting. The truth however is that unless fundamentals are fixed, getting to insights and leveraging the latest techniques is incredibly tough. But what are these fundamentals? There are three key aspects:
- Clean and reliable data sources. Having a single source of truth (golden data tables) for at least the most often used or most critical data sets, e.g. revenue, product usage, and sales funnel.
- Governance. At least the golden datasets should be maintained, governed, have SLAs attached, lineage tracked, data dictionaries available, and offer data contracts for users.
- Robust processes. Clear, defined and documented roles and responsibilities, and runbooks across teams producing data, processing data and consuming data.
Here are the 3 things companies can do to build these fundamentals:
- Limit the distractions
We are living in times of rapid technological innovation and the need for quick results. Every time a new advancement happens there is an inevitable question of how one can use it to create a competitive advantage. Take for example the recent excitement about ChatGPT. Undoubtedly a pivotal technology. Maybe it can help a new hire engineer ramp up quickly by understanding existing code easily, or create efficiency by fixing bugs or writing test cases. But it wouldn’t help to understand why the cost basis created by the product and finance team differs, which one is right to use for pricing decisions, which in turn will impact revenue guidance to the street. Sometimes these technologies can be a distraction if the data maturity of the company is low.
One can limit distractions by focusing on achieving outcomes in the simplest possible way. A perfect example of this would be trying to predict the success of a particular movie through analysis of its emotional arc. While the advanced NLP techniques are good at surfacing which arcs perform well in general. A simple average of the past IMDb ratings of the lead actors and director of the movie is a much better predictor of success for a particular movie.
2. Do a ring-fenced and steel-threaded fundamentals build
Fixing fundamentals is usually a massively cross-functional and multi-year effort. This can be daunting but two approaches can help solve for it:
Ring-fenced capacity. The objective here is to have guaranteed capacity and prioritization. There would usually be multiple teams involved in fixing things end-to-end: platform/infrastructure, data engineering, business intelligence, data science/analytics, functional teams, data governance etc. It is critical that fixing the fundamentals is a priority for all teams and they either dedicate certain person(s) for the project or guarantee capacity e.g. a number of story points if sprints are used.
Steel-threads. This is a critical one for success. Simply said, at the start, build everything end-to-end but for just one use case. The reason it is a ‘steel’ thread is that it should not break and let any element slip off. And the ‘one’ use case is important because if one tries to fix all fundamentals overall it becomes counter-productive, looking more like a large-scale IT project. Focusing on one use case helps deliver value to the business quickly, get learnings and create a win. While each use case will be unique but one steel thread followed through will create templates and establish capabilities that can be built upon. E.g. Model documentation for one use case creates a framework of how to do the documentation. Similarly, extensible capabilities get built e.g. data lineage tooling. Below is a starting point example of using a churn prediction model as a use case, and the elements to be “steel-threaded”, the questions to ask, and the capabilities to create.
3. Focus on enablement
Ever wondered why people in your company don’t use data or are interested in that cool tool you built? Maybe they just don’t know how to use it. Or maybe they want to, but feel it would be too time-consuming. Or it could be a simple matter of hesitation to ask for help.
To make good use of data the whole organization needs to be enabled. Everyone needs to come along on the learning journey. Data teams need to learn storytelling and functional teams need to learn data story listening. And again here the ‘fundamentals build’ I spoke of matters. Pick one use case that really matters for the business team and run it to the ground. E.g. A sales team might care about what products a customer is most likely to buy. Enable by creating a reliable transformed data set, building a model on top of it, creating a tool/dashboard that helps sales teams get insights easily, doing training sessions, and creating governance artifacts.
Building fundamentals is hard work but the impact is also big. Once a steel-thread gets built, I’ve seen exponential progress happen on use case builds. This opens doors to many possibilities and the organization finally starts to believe that it can use data to create a competitive advantage after all. And that feels even fancier than fancy AI.
