How Companies Can Stop Failing at AI and Data-Driven Decision-Making
Four levers can help business leaders succeed in making the best use of data
Published in
7 min readFeb 8, 2023
Data has been a hot topic for over a decade now. It took off with big data and over the last few years, AI and machine learning have come to the forefront. The last decade was also a decade of immense growth. Capital was aplenty and companies had long runways to learn and perfect their craft. Yet research points to the failure of organizations to leverage data effectively. The economic climate has now changed and companies are now more focussed on profitability and operations. The cost of failure is too high now and companies thacanto leverage data will have a competitive edge. AI and data-driven decisions can be of help here by bringing clarity to discussions, separating facts from opinions, and accelerating time to market.
However, most companies have not set up an organizational model that is geared towards AI and data-driven decision-making. As companies pivot strategies for the new economic realities four key elements can help build a data-based decision-making culture:
Creating clarity on why and how data needs to be used
- Aligned Purpose, Goals, and Metrics. Today’s workforce is highly purpose-driven. To them, efficiency, profitability, growth, etc. are a means to an end and not the other way around. So the leadership needs to be very clear on the purpose the company is driving towards. That purpose then needs to be translated into specific goals, and then further distilled to exact metrics that need to be measured. One of my consulting clients, a roadside assistance company did this particularly effectively. Their purpose was to ensure that no customer was stuck on the road for more than a certain time depending on the service zone. This was then converted into a goal for each organization, for example, Finance budget allocations were optimized to achieve this objective, Customer Service would aim for first-call resolution, and Sales were incentivized not only on target achievement but NPS, so that they sold the right products to the right customer. Finally, the executive dashboards had all these metrics and in the annual planning process targets were set for these P0 metrics.
- Clear data-oriented problem statements cascaded through the organization. Having a clear set of problem statements brings alignment across the entire organization. These need to be simple at the company level e.g. “Increase operating margin by 50bps”. This helps everyone align to what needs to be solved for and puts a data point behind it. The overall goal can be then disaggregated for functions and they can decide what levers they want to pull. For example, marketing might create a problem statement “Change media mix model to maintain MQLs while reducing marketing spend by $xx”
- Executive “use” of the data and not just “buy-in”. Oftentimes even when executives agree that data-based decisions are essential, they continue to rely on their gut. Executives should take data-based decisions even if sometimes this is against their gut. And they should communicate this as well so that the entire organization feels that they are a data-driven organization. One of the most effective sales executives I have worked with understood the power of data and ensured his team made the best use of companies data capabilities. Two of his go-to methods were to never make a decision without having at least some supporting data and framing very precise statements for the data team, for example — “I want to understand the impact of 5% and 10% price increases on our volumes and margin.”
Having a shared understanding of the data
- What is being measured? We often perceive data to be an objective thing, but in reality, it can be very subjective. Organizations often have very fuzzy and sometimes diverging definitions of even fundamental metrics such as growth, churn and cost to serve. A shared understanding of what metrics are to be used to measure the organization and their exact definitions is a must. Lacking this, data becomes an igniter of debate rather than a tool to drive decisions. I have found that one of the most effective ways to achieve this is to have a company-level data dictionary with precise definitions and data sources.
- Who owns the measurement? We live in the times of matrixed organizations, where several teams work cross-functionally, often with slightly overlapping mandates. If multiple people or teams own a single thing, no one ends up truly owning it. Strong ownership and process are paramount. Everyone should have clarity on who owns and provides each metric. While there can be debates around what is the right way to define a metric, once debated, the number that the owner of the metric provides is the truth. Similar to the point about, having an owner identified in the corporate data dictionary is paramount.
- What are the golden sources of truth? It is not uncommon to find more than a dozen sources of truth in a large organization. Every team creates its version of the truth, which suffices its use case. This is a recipe for creating confusion in the organization. Often this is considered a tactical data topic, whereas it is an executive leadership topic. It is essential to have a set of golden tables that house the core data such as financials, go-to-market metrics, and people metrics. This is one of the biggest causes of lost efficiency in analytics, with analysts spending countless hours trying to reconcile data, and executives spending precious time debating the data instead of formulating strategies to succeed in the market.
Building foundational elements
- Simple tech stack. The simpler the technology, the easier it is to maintain, easier to explain, and easier to use for the entire organization. Engineering teams lean towards shiny objects, but the business goals and the use cases should define what technologies need to be implemented and not the other way around. Technology today is moving at a very rapid pace and innovations abound. This has led to situations where we have hammers looking for a nail. I have seen this leading to numerous organizations having multiple and complicated data stacks with little interoperability. It is best to keep the stack as simple as possible, sometimes Power Pivot running on top of a database is all one needs.
- Self-serve analytics. Data has become so vast today that unless contextualized, it becomes tough to make good use of it. This context usually sits in the business and the specific functions. The data infrastructure needs to enable analysts and even business folks in functions to make use of data. Tools such as Alteryx can make it easy for non-technical teams to build their ETLs, and tools like Tableau can help them make their reporting repeatable. This is not only a question of technology, but of culture, and the philosophy of how an organization uses data.
- Governance and processes. These are usually the last areas of investment in an organization. It is a topic that does not get much love from either executives or the data teams. In time though it becomes one of the limiting factors on speed. For example, it is common for new hire analysts to take months to ramp up as they wade through the messy data; and still, spend weeks finding and cleaning data for an analysis that should take a few hours.
Creating a culture where data is valued and used
- Empowered decision-making. Decisions are not the sole prerogative of leaders and should not be. The micro-decisions finally add up. There are hundreds of decisions that can take place in an organization every day. E.g. Pricing decision on a small deal, software plug-in selection for a process workflow, prioritization of backlog, etc. Clear decision rights and a culture of using data to inform can lead to massive efficiencies. Employees feel empowered and can move quickly. But this is hard to achieve, and in my experience, it is an organizational transformation topic that needs to be leadership driven and highly cross-functional in nature.
- Define rituals and artifacts. Cultural changes need to be lived and driven through action. Fostering small changes can have a big impact. One of the most effective ways to do so is by creating rituals and embedding them into ways of working. A simple but powerful one could be that if a decision is needed in a meeting the requester needs to send analyzed and aligned supporting data before the meeting as a pre-read. Similarly, I have observed, starting operational reviews by looking at a set of metrics is highly effective in decision-making. Amazon’s memo and press releases are great examples of such artifacts.
- Connect data and business. Whenever there is a transactional relationship between data and functional teams they lose the insights in translation. Joint business+data teams that are working towards the same problem statements are most effective. Having clear problem statements such as “How can we improve conversion rates with the same marketing dollars?” can help align teams to the same goal. Then the business teams need to coach data teams on the context and data teams need to coach the business teams on the art of the possible but also the limitations of analytics techniques. Data and technology teams need to have a seat at the table to be effective. Organizations where they are a backend function lose on squeezing the value out of their data.
Any type of organizational transformation is difficult, and becoming an AI and data-driven decisioning organization is even more so as it needs to bring together the elements of business, technology, and data analytics. However, it is possible, and using some of the tactics discussed in this article can provide a head start.
