Data Science in the Real World
Call to Data Scientists — Stop using Measures like Accuracy, Precision or Recall!
Communicating in a language decision-makers can understand
Okay this title is deliberately provocative and dripping with hyperbole. As one of our data scientists said “I was appalled by the title but completely agreed with you in the end”.
I’m a strong believer in the opportunities machine learning and the field of data sciences offers but we have to be honest with ourselves. There are growing pockets of skeptics, inflated expectations, and some are even warning of a looming credibility crisis in data sciences.
In addition, many decision makers do not have the proper training to help interpret, understand and properly apply the output from our models.
COMMUNICATING WITH IMPACT
To get ahead of this, we need to better frame our impact to key stakeholders and decision makers. In the end it’s important that impact is conveyed in the context of the business problem and in a language non-data scientists can understand.
“Focus framing your model results in the context of the business problem”
For example, I don’t think measures like accuracy, precision and recall are particularly useful metrics for most business users. In certain projects with unbalanced datasets, we’ve had single digit precision numbers that have led to substantial business impact especially when contextualized against the process the model is replacing.
This is the key point — we need to make sure we are comparing results against how things are currently being done.
Key business stakeholders are often reluctant to implement or even pilot a new model unless they can clearly understand how it could drive significant impact against their key business metrics.
HOW TO BEST FRAME YOUR RESULTS
When possible, it’s in our best interest to ensure we are framing results in a business context that our end users can understand. Let’s imagine you are meeting with the VP of marketing to present results from your model for diagnosing patients with a rare disease.
Look at the progression below as we translate model results into information that helps support an informed decision from a key stakeholder:
I admit this is a simple example and there are other considerations to take into account, but consider this illustrative to hammer home the point.
In the context of the business, it should be about quantifying the improvement vs the current process.
This brings up two important concepts — ‘fit-for-purpose’ and the idea of a challenger model. In the ideation phase of an analytics focused project, we employ the ‘fit-for-purpose’ design principle. Before jumping right to an advanced ML model, ensure you are exploring simpler approaches first. Where possible, think about deploying a simpler approach first (i.e. challenger model) to see if that can achieve desired business outcomes.
WHAT ARE YOUR STAKEHOLDERS NEEDS?
The key point to remember is that your decision makers may have varying needs which drives different perspectives on metrics that matter. Some may be focused on increased sales, others on decreasing costs, others may be focused on reducing errors while others just want you to make their job easier.
“Avoid unnecessary model rot — define success in terms of your stakeholder’s needs”
Ideally these metrics that matter are discussed prior to any project and in collaboration with your key stakeholders. Defining what success looks like in the language of your stakeholder can be a major factor in driving meaningful and sustained business adoption versus your model being left to rot on the shelf.
Special thanks to our two talented data scientists Emily Kogan and Erik Sjoeland for contributing to this article.
