Quantify the Business Impact of Your Data Science Project
Get Your Work Valued & Recognized…
Have you ever shared a room with business stakeholders to try to justify the impact of your Data Science (DS) project? Have you encountered the question, “What is the estimated ROI on your project?” Was your DS project ever dropped from a portfolio because its business impact was not identified or significant? Most of us have faced similar situations once or more. At least, I have been through such a dilemma. My compliments to those lucky folks who never had to respond to such perplexing questions!
The truth of the matter is that we — i.e., the DS or Data Analytics community — always struggle to quantify the impact of a DS project. Over the years, I have encountered many challenges in this domain. However, I always transformed my methods through experimentation, which has served me well to accomplish my tasks. Therefore, I thought to share my experiences with you. I am not claiming to offer a complete solution to all your DS project issues, but my article will give you a quick start. I hope that article will perhaps generate a healthy discussion, in which all of us will share valuable ideas, and we can learn various solutions together.
Before we discuss the impact of a data science solution, the fundamental questions you must always inquire yourself are the following: “Have I chosen the right business problem with real and meaningful (+ ROI) impact on the business?” “How do I determine if I (along with my team) am working on the right business problem that has a meaningful impact on our company,” and “Where do I start?”
Let’s be realistic here! If you are solving a problem that saves a few hours of time for your teammates, or it automates a manual task in a process that does not produce any considerable business outcome, the impact of your DS project may be limited and inconsequential.
Primarily, it would help if you always considered that the problem you are solving is aligned with your company’s objectives. For example, suppose your company is manufacturing (or developing) a product or providing some services, and it cannot yield enough products to meet the customer demand. In that case, the strategic objective of your company automatically transforms to improve its production yield, reduce product waste, add innovative & advanced facilities and resources to build more products etc. In this scenario, we can create a DS solution that improves the yields and reduces product waste.
You may ask, “Well, what if the company objectives are discussed at a very high level, and I cannot associate the business problem at hand to the company level objectives.” In this case, you can strive to link the objectives one level down — let’s say, at the business functions level. This level includes R&D, Manufacturing, Operations, Sales & Marketing, Supply Chain, Customer Service etc. Here, we can assume that all the underlying business function objectives and processes should be aligned with the overall objectives of the company. Nonetheless, you may also consider even descending to the next level of organizational structure to specify the objectives.
Another advantage of linking your DS projects to a company’s objectives is to bring business stakeholders on your side. If they can see any value of your project aligned with their company’s objectives, they will consider you their associate. I cannot emphasize enough to inculcate that it is critically significant to have a business sponsor (i.e., stakeholder) for your DS project. These sponsors will help you acquire the necessary funding and change management related to any roadblocks. Moreover, they will facilitate the adoption of your DS solution in their departments. According to my experience, most projects fail before implementation due to mismanagement of various relevant activities, such as onboarding, awareness, training, post-implementation support, etc. Here are a few Key Learning points for you:
Key Learning #1: Adopt a top-down approach and start with the objectives of the company or business function or business sub-function. Here, the idea is to identify those DS problems that can be aligned with the company objectives at some specific level. It has been be observed that these problems have a broader and great influence on the organization.
Key Learning #2: If you cannot identify the business outcome of your DS project, it isn’t very worthy of working on it. Therefore, you should proceed to explore another problem on your to-do list. If you still proceed to work on a problem without specifying its impact, you might be surprised at the end.
Key Learning #3: It is more convenient and easier to persuade your business sponsors if they can see the success of their projects through your DS solutions.
Key Learning #4: The availability or absence of a business sponsor/champion for your DS project can determine the fate of your solution. Therefore, if you cannot find a sponsor, you must think twice before investing your efforts and energies in developing a solution.
Ok! Now, your next logical question should be: “What metrics should I define to quantify the business impact of my DS solution?” In the table below, I have added various dimensions of impact (or outcome) to consider your DS project. You can evaluate how one or more of these dimensions can add value to your data science project. The KPIs and metrics given within each dimension have been mentioned as examples only. You can promptly use these metrics or KPIs for your own project scenario: (Definition metrics: Measure where KPIs: Quantify with target).
Cost Reduction: There could be multiple variants of cost reduction metrics. For instance, raw material cost reduction per unit, cost of goods sold per unit, cost of goods manufactured per unit etc.
Yield Improvement: Yield can be improved by measuring various process parameters at critical production phases and adjusting them to acquire optimal production; for example, measuring the percentage of waste/rejection reduced.
Human Time Saving: If a DS solution can save manual labor time through automation, it is beneficial for a business. Some examples include: reducing manual labor by 30% per day, per batch, per sprint or any other suitable metrics.
Process Improvement: If your DS solution can help improve a business process, it can save time and costs. For example, reduce cycle time by 20 hours, reduce waiting time by 2 hours, reduced exit cycle time by 8%, and minimize the customer return processing time by 6%.
Speed: A data science project can improve the completion speed of a process, which can be worthwhile for a business. Speed and process improvement can be overlapping metrics, and they can be combined. For example, reduce time to market by 100 days average delivery time by 10 minutes.
Better Quality: Many metrics can quantify the overall quality of product, product health, customer services, customer experience; e.g., reduction in number of customer returns by 20%, decrease in number of customer complaints by 14%, reduction in warranty calls by 10%, and decrease in number of bugs by 12%.
Market Share: The success of a DS project can be quantified through its impact on market share. For instance, growth in market share by 5% compared to the previous year for a given brand, geographic region, customer segment or overall consumer base.
More Customers: Improving the number of customers is also an imperative outcome for a DS project. For example, growth in customer base by 6% year over year, increase in average time spent per user session by 10%, increase in customer conversion rate by 25%, and increase in customer retention by 12%.
Now you must be curious to know: “How do I utilize these metrics in my project?” To answer this question, let us put the process of using them into the following steps:
Step 1: Choose the right metrics for your scenario. Schedule a discovery sessions with business users and walk them through the metrics dimensions listed above. Determine which metrics will apply for your use case.
Step 2: To quantify the impact of your DS solution, you must establish the baseline for each metrics. For instance, if your company is dealing in semiconductor products, and your current average production yield is 94%, it implies that 6% of your products are being rejected. These 6% products could either be defective or they could be False Rejects — i.e., good products rejected during the quality inspection process. We can create a scenario here by stating that if 20,000 products have been produced per day, then 1200 of them (i.e., 6%) have been discarded. Hence, 94% (or (18800/20000)x100)) is your baseline, and you want to measure if your DS solution could improve the yield or not. However, your company may or may not maintain the historical raw data to come up with the baseline value. Suppose you cannot acquire the readily-available historical data. In that case, you might need to work with multiple business teams or users to make assumptions and approximate the raw numbers to come up with the required baseline value. Hence, your numbers do not have to be accurate, rather a high-level approximation.
Step 3: To measure the impact, you need to wait for certain period after deploying your DS solution and then collect the new raw data for the same metrics in Step 1 (i.e., Average monthly units produced and rejected). Alternatively, you can compare cumulative data of , say, six months before and after the implementation of your DS solution. I reiterate here that you must wait for a reasonable period (e.g., ~6 months or whatever is relevant to your use case) to witness the real impact of change.
Key Learning #5: If you do not have any raw data to compute the baseline, make assumptions by discussing with your team members who have intimate knowledge of the business process your solution is implementing. After all, as leaders, we always make assumptions, if we do not have the full information available.
Key Learning #6: You need to wait for some time (typically six months) to see the impact of your DS solution.
Key Learning #7: The business value calculation is not a simple process, and it requires perseverance and patience. To implement your DS solution, you need close collaboration with team members working in business processes. Consequently, it would be best if you were iterative and diligent in evaluating your DS solution’s business value or impact.
Now, I elaborate on the most challenging part. How can you evaluate whether your DS solution eventually provided the expected outcome? There could be a scenario that some other concurrent initiative by the company might have achieved the same outcome you were targeting. Furthermore, in some cases, your DS solution may provide some insights to be utilized by managers or other folks to get an outcome. In that case, can you attribute the project’s success to your solution or to the individuals who took actions to make things happen at the right time, or is it a mixture of both these scenarios? I reckon we can split the attribution of success by 50–50 or some other reasonable ratio. Unfortunately, there is no scientific methodology to address this issue, as it highly depends upon a certain situation. Hence, it would help if you worked with other participants to assign this attribution ratio — as agreed by all stakeholders.
Key Learning #8: Timing is everything. When you are ready for DS solution deployment, it is imperative to be prepared to attribute the impact of your solution. Always think broader at an enterprise level to evaluate the possibilities of other initiatives impacting the outcome of your company, as they might also be working towards the same solution like yours.
Any questions and feedback are welcome! If you deem alternate ways to quantify a DS project outcome, please share them in your feedback comments.
