Most data scientists naturally gravitate towards the fun parts of Data Science – developing a technically advanced, sophisticated machine learning model. However, a good chunk of data science managers invest too much time in a model’s technical design, and don’t invest enough time in developing deeper understanding of the business problem the model is intended to solve. As a result, technically successful projects are deployed into production, only to fall short of delivering the anticipated business value and taking it’s permanent home in what I like to call the data science graveyard
. As a data science manager of a large team, I’ve buried more projects in that graveyard than I care to admit. However, these experiences have taught me valuable lessons about ensuring data science projects generate real business value. In this article, I’ll share the four important lessons that data science managers can use to ensure that data science projects generate clear and meaningful business impact.
1: Align Projects with Company’s Business Goals
Most, if not all, companies share strategic goals and objectives on some regular cadence. An example goal might look like "increasing customer retention X% by Y date". These goals and objectives represent a company’s operating north star. They clearly prioritize different parts of the business and help employees make decisions about what is important to work on and what’s not. With that being said, every data science project your team undertakes should be directly linked to a specific business goal or objective.
Achieving Alignment in Practice

On my team, we’ve implemented a process that ensures our projects are tightly linked to the company’s strategic priorities. At the start of each project cycle, we get together as a team to review our running list of things we could work on using the tools and data we have available. We take two passes at this list – the first focused on which projects have the potential to impact a specific company goal or objective, and the second focused on quantifying the business value a project could create. At the end of this process, we’ve effectively prioritized our backlog to make sure the most impactful projects are correctly sitting at the top of our backlog and the least impactful projects claim their rightful place at the bottom of the list.
This process takes less than an hour, and guarantees the team is spending time on the projects most likely to generate meaningful value against our company goals and benefits. Additionally, it helps to advance the team’s knowledge of goals and objectives beyond what is shared from the top down. This improved knowledge strengthens the team’s ability to make good decisions throughout the project and ensure we don’t sacrifice business value unnecessarily.
2: Engage & Collaborate with Stakeholders
Effective stakeholder engagement is crucial for ensuring that data science projects are well received and impactful. Without consistent, regular and effective communication there will eventually be a divergence between a data science project and the business value it’s designed to create. With that being said, managers play a key role in bridging the gap between data science teams and business units, facilitating communication, and ensuring both sides remain aligned.
Engaging Stakeholders in Practice

In my current role, I don’t follow a rigid stakeholder engagement process. However, I’ve identified four key components for effective stakeholder Management.
- Identify Stakeholders Early: Data science projects are often part of larger product updates or initiatives, meaning that what’s built can affect multiple teams – whether that’s engineering, product, or a specialized business unit. Ensuring that the perspectives and viewpoints of all relevant stakeholders are considered early on makes it easier to create solutions that are applicable to the problem statement and generate real business value.
- Agree on Success Metrics: As a manager, you can help stakeholders to understand how the model will be evaluated during its development. Furthermore, you can help these same stakeholders to understand how the metrics used to evaluate the model connect to the problem statement and company goal the project seeks to impact. Ensuring stakeholders all have a solid understanding of what success looks like means that once the model is live, it will be easier to measure and share success. Achieving consensus should be collaborative and often times, stakeholders will help the data scientists to refine their success metrics so that they correctly measure the impact the model has on the business.
- Consistent Check Points: In my experience, data scientists sometimes overlook involving stakeholders during the technical aspects of the development process, often only engaging them to confirm the problem statement or share final model results. However, stakeholders can provide valuable insights during stages like setting up training data, engineering features, and evaluating outcomes. Don’t hesitate to collaborate with stakeholders just as you would with your data science team.
- Be Open To Feedback: Undoubtedly, stakeholders will have a lot to say as you share updates and progress with them. It’s easy to dismiss what they have to say as a lack of technical understanding or inexperience working within data science. However, the reality is that these stakeholders often represent perspectives of the end users of your model. If you want your model to aid the business and generate real business value, listen to what these different stakeholders have to say!
3: Measure Success with Clear Metrics
To ensure that data science projects deliver real business value, it’s essential to establish clear metrics for success. Without clear metrics for success, even the most technically impressive models can end up irrelevant, underutilized, or fail to drive the desired outcomes.
Establishing Clear Metrics in Practice
Early in my career, this is something I struggled with almost perpetually. I had a hard time bridging the gap between complex loss functions and the business value we were trying to create. For that reason, I share some traps I often fell into when defining success metrics and how to avoid those traps in practice.
- Focusing Exclusively on Technical Metrics: There is always a tendency to focus on metrics used during model training, such as accuracy or F1 scores. These metrics are key to a data scientist’s ability to build a strong model, however they shouldn’t be used in isolation. For example, if a project is intended to increase customer retention – then even a highly accurate model might not be useful in practice if customer retention stays the same after the model has been deployed. With that being said, when defining success metrics for a project, ensure that those metrics encompass technical performance as well as the model’s impact on key performance indicators that captures the business value you seek to create.
- Ignoring Stakeholder Feedback: I alluded to this in a previous section, but stakeholders often have a lot of valuable feedback that can be helpful in defining and refining metrics so they better align with business goals and objectives. Ignoring this feedback can lead to a misalignment between the model’s performance and it’s ability to generate real value for the company. As I mentioned previously, engage with stakeholders early, work together to define the right metrics, and share these metrics with the stakeholders consistently.
- Overcomplicating Metric Definitions: In my opinion, simpler is always better. There is no reason to track a plethora of different metrics, or define metrics in such a way that it takes a small team to understand why the metric is moving in a certain way. My advice to all managers is to keep it simple. Your metrics should be easy to communicate. Don’t be afraid to focus on one or two metrics that capture the essence of the problem you’re trying to solve.
4: Continuously Monitor and Validate Performance
At the start of this article, we talked about the need to ensure every data science project is aligned with a company goal or objective. However, without the right monitoring in place you will never know if the model is helping to generate value or not. More specifically, continuous monitoring of model performance against established metrics is a crucial step in confirming that your data science project is working as expected, and generating value over time.
Monitoring in Practice
In practice, monitoring frequently gets overlooked. It’s easy to point to the need spend time solving other problems as a reason to not do it. However, monitoring capabilities can be quite easy to implement. Here are some practical steps we take in my current role:
- Consider Monitoring Early: When developing a data science project plan, monitoring should always be included. In every project my team takes on, we explicitly allocate time to stand up monitoring capabilities in our project plan. Thinking of it more as something that has to get done, instead of something that should get done will make it easier to justify working on it before considering the project complete.
- Focus on Defined Success Metrics: Monitoring doesn’t mean reporting on every model metric under the sun. Instead, your monitoring framework should focus on the agreed upon success metrics that have a clear link between model and problem statement/company goal. Since, hopefully, these metrics have been shared and validated with relevant stakeholders, it will be quite easy to quantify the value created by the model.
- Automate Regular Performance Checks: Instead of monitoring model results manually and sporadically, set up automated systems to collect the required information and organize it in the way needed to build the monitoring. Whenever I say this to anyone, they assume it means a cross functional effort to stand up intricate and complex systems that monitor predictions in real time. However, there are tools you can use to achieve a simpler version of the same thing. For example, Github Actions is a great way to do this – just set up a python script to get the required data and use Actions to execute this script on a timer. This script can send the data to a database for dashboard development or simply sent to a spreadsheet. The point is that this requires little to no support from anyone besides the contributing data scientists on the team.
Delivering real world business value through data science projects requires more than just technical expertise. As a data science manager, it’s your responsibility to ensure that your projects not only succeed technically, but also align with your company’s strategic goal, and consistently generate measurable outcomes. As you move forward in your data science journey, I encourage you to incorporate these practices into the way you run your teams.
Help me grow my page!
All claps and comments are appreciated. It’s how Medium knows if I’m doing a good job!
- Follow me on Medium too
- Subscribe to my newsletter below