Model Interpretability

Using machine learning to automate processes is widely adopted in many organizations worldwide. As a part of the Data Science Factory team at Dell, we work with different business units to deliver innovative data-based solutions to optimize existing processes.
In one of the engagements we had with the pricing business unit, we were focusing on improving a manual intensive process that led to many delays. The manual process included auditing deals submitted by sales reps (where some of the products were priced below a certain point) and deciding, based on the characteristics of the deal, whether to approve or deny it (you can read more about it here). In cases where a deal is being denied, the pricing analyst should provide an explanation to the sales rep. Providing an explanation can help the sales rep adjust the deal in such way that it will get approved, which will be beneficial for both sides.
Since it is very time consuming and can lead to lost opportunities, we wanted to optimize it by reducing the volume of deals that needs to be manually reviewed. To achieve that we used machine learning to auto-approve and auto-deny the simpler cases, this allows the analyst to focus on more complex deals which will generate more value.
In order to keep the same level of quality, and to create trust among the users (which is very critical when implementing machine learning models), we had to understand how decisions are being made. With that in mind, I sought for existing tools to support the transformation we were doing, and I came across Lime and SHAP. Throughout the process, I examined both methods on the data I had to understand which one will be a better fit, and I gained a lot of knowledge that I wanted to share. In this blog post I’ll share what I’ve learned and share some details that helped me get a better understand of how each tool works.
This blog post will cover the following:
- Shap & LIME- background
- Examples
- Summary
- Further reading
1. SHAP & LIME- Background
LIME (Local Interpretable Model-Agnostic Explanations) provides local explanations; it can help explain why a single data point was classified as a specific class. LIME treats the model as a black box; it doesn’t distinguish between random forest, a decision tree or neural networks. It uses linear models to provide a local explanation. The main concept around LIME is that even though linear models won’t be able to explain well complex data/model, when trying to explain an observation locally it can provide an adequate explanation. Below you can see a figure explaining the intuition around LIME. this was taken from the paper: "Why Should I Trust You?" Explaining the Predictions of Any Classifier.
LIME offers three different explainers: tabular explainer (which will focus on), image explainer and text explainer to handle different types of data.

SHAP (SHapley Additive exPlanations) aims to explain the prediction of a single data point by computing the contribution of each feature to the prediction. The generated explanation is based on computing Shapley values from coalitional game theory (you can read more about it here). It is based on the following scenario: let’s say you play a certain game as part of a group, the group wins and as a result earns some money (payout). What is the best way to divide the payout between the players, to best reflect the contribution of each player to the game? In our case we would like to understand how to divide the prediction (payout), between the different features (players) in the model to best reflect the contribution made by each one of the features.
In general, to calculate the contribution of a certain feature to the prediction, you should get all the subsets that do not contain the feature in question and compute the difference between the predictions before and after the feature was added- then average out the different subsets.
The described process can be computationally expensive, plus running a model without a certain feature creates a completely new model, which is not what we want to explain/understand (you can hear about it more in this great lecture given by Adi Watzman). Luckily, SHAP uses different approximations and optimizations that overcome these difficulties.
SHAP provides three different explainers: KernalSHAP, which similar to LIME is model agnostic, TreeSHAP which is optimized for tree-bases models, and DeepSHAP which is a high-speed approximation algorithm for SHAP values in deep learning models.
In this post, I’ll focus on the tree explainer since we used a tree-based model in the project we’ve done.

2. Examples
Moving on to some examples bases on our use-case, using LIME and SHAP.
LIME
To provide an explanation, first we have to create an explainer. The explainer should get the following:
- We should specify whether we’re dealing with regression or classification model.
- We should pass on our training data, features names’, and class names (this is optional, the default is 0 and 1).
- Verbose- indicates if we want to present additional details with our explanation.
- Since there’s some randomness in the process, to be able to reproduce the results, you should set up the random_state variable as well.
from lime.lime_tabular import LimeTabularExplainer
lime_explainer = LimeTabularExplainer(train_data.values,
mode = 'classification', feature_names = new_feature_names, class_names = ['Deny', 'Approve'], verbose=True,
random_state = 42)
A side note regarding categorical features– if your model contains categorical variables, it complicates things a bit. Here’s a short explanation: in order to provide explanations, LIME uses the original dataset to sample data points around the observation we would like to explain. If we’ll provide the data where the categorical variables are in a one hot encoding format, the sampling process can create samples that are unlikely to occur in the data, and the generated explanation won’t represent the actual data. To over come that, the categorical features should be transformed into integer labels, to learn more about how to use LIME with categorical features, I encourage you to watch a great lecture given by Kevin Lemagnen or visit his GitHub page.
After the explainer is all set up, let’s generate an explanation. To get an explanation, you should provide the following:
- The instance you would like to explain.
- The function that generates the probabilities (in case of classification models), or the function that predicts (in case of regression models) for our data set (in our case- predict_proba).
- You can also specify the maximum number of features you would like to use to build the local linear model, currently we’re using the default value which is 10.
exp = lime_explainer.explain_instance(instance, trained_model.predict_proba)
exp.show_in_notebook(show_table=True)
Next, lets examine the output:

At the top, the intercept of the linear model created by LIME is presented, followed by the local prediction generated by the linear model, and the actual prediction from our model (this is the result of setting verbose in the explainer to True). As you can see, the prediction generated by the linear model and the result generated by our model is quite close. Next, you can see the probabilities for each class, as the original model predicted.

On the far right, you can see the values of each feature for the specific observation. The features are color coded by the class they contributed to; the features colored in orange contributed to the approval of the deal, while the features colored in blue contributed to the denial of the deal. Plus, they are sorted by their affect on the prediction, where the feature that impacted the most is at the top. In the middle chart, you can also see the magnitude of each feature in each class. It’s also important to understand, that for LIME to provide more intuitive explanations, the numerical values are being discretize into several groups, that is why in the middle chart the features are provided along side a certain range. This is the default in LIME, and it can be changed by setting the variable dicretize_continuous to false.

You can get more information on the local model that was generated by running the following code:
print('R²: '+str(exp.score))

This will give us the R squared of our local model, we can use the score to decide whether we can trust a certain explanation or not.
Additionally, you can access the weight given to each feature in the model with the following code:
exp.local_exp

Summing up the weights and the intercept will result in the local prediction generated by LIME.
exp.intercept[1] + sum([weight[1] for weight in exp.local_exp[1]])

SHAP
Moving on to TreeSHAP, lets generate an explanation for the same instance we used for LIME.
First, similar to LIME, we should import SHAP, create the explainer and pass in the trained model:
import shap
shap.initjs() #This is for us to be able to see the visualizations
explainer = shap.TreeExplainer(trained_model)
Next, lets generate an explanation and visualize it. The first line calculates the SHAP values for the instance we provided, and the next line generates a plot that will visualize the contribution of each feature to the final prediction. The calculation made by SHAP are made for both classes, and here you should choose the class you would like to refer to using the index (in regression models it is not needed), in this case I would like to explore the results for the approval class (class 1).
To calculate the SHAP values for the instance we want to explain, you should provide the instance itself, and to generate the plot you should provide the following:
- The expected value generated by the explainer, which is essentially the average of predictions on the training set, it is also noted as the base value.
- Next you should provide the SHAP values calculated in the previous line (these are the contributions made by each feature to the model’s prediction).
- The instance you want to explain, and the features names’.
shap_values = explainer.shap_values(instance)
shap.force_plot(base_value=explainer.expected_value[1], shap_values[1], features=instance, feature_names=new_feature_names)
The bold number in the chart below, is our prediction. On the left, you can see the base value, which as explained before, is the average of predictions in our training set. As the name states, this is the value our prediction starts from, adding up the contributions made by the features in our model, will result in our final prediction.

The features are arranged in a way that all features that contributed to the approval (class 1) of the deal, are on the left and colored in red, and all the features that contributed to the denial (class 0) of the deal are on the right colored in blue. Notice that the features on each side, are sorted by their magnitude, which is also apparent in the space each feature gets. You might also notice in the edges of the chart, there are additional features that also contributed to the different classes, but their contribution was significantly smaller than the ones being presented, you can hover the edges in the plot and you’ll be able to see the features’ names and their values.
TreeSHAP utilizes the structure of the tree, and can calculate the exact values an not an approximation. Summing up the expected/base value will generate the exact prediction, as you can see below.
explainer.expected_value[1] + sum(shap_values[1])

SHAP also provides different visualizations that can give you an overall overview of each feature’s behavior based on the class you choose. To do that, first you should calculate the SHAP values for the entire test data (be aware that it might take some time), let’s look at an example:
test_shap_values = explainer.shap_values(test_data.values)
shap.summary_plot(shap_values[1], test_data.values)

Let’s examine the different aspects of the chart:
- x-axis represents the contribution to the prediction, these are the SHAP values.
- y-axis on the left displays the most impacting features (you can specify the number of features you would like to display- the default value is 20). The features are sorted by their effect on the prediction, where the most affecting ones are at the top.
- y-axis on the right, indicates how to interpret the original values for each feature. In each line, the data points are colored in either blue or red, blue indicates low values of the feature, and red indicates high values of the feature.
- The plot is separated by an horizontal line, where in the right side you can see data points the contributed to the approval of the deal , and on the left side you can see data points that contributed to the denial of the deal .
- For example, low values in feature x_29 will contribute to the approval class, while high values will contribute to the denial class.
This can help you understand if the features are behaving like the SMEs (Subject-Matter Experts) were expecting them to behave, and it can help us find data related issues that we didn’t noticed before.
There are other options for global visualization in SHAP, check out the author’s GitHub page to see the different options.
3. Summary
In this blog post I shared with you my motivation of using tools to explain ML models, we reviewed the basics of LIME and SHAP with examples based on our use case.
Looking back at the work we’ve done, I feel that the use of these tools generated trust among our business partners, which led to a much smoother way towards implementation of the model we built. Furthermore, using these tools did help us catch some data related issues that was fixed consequently.
LIME and SHAP are both great tools you can use to generate trust as well as to give explanations for a certain decision that was made. When we examined both on the same instance, we’ve noticed that most effecting variables are pretty much the same in both LIME and SHAP. However, the order is a bit different between the two.
The main advantage of LIME is that it’s faster (at least for tabular data), and the way it works is pretty intuitive. However, the generated explanation sometimes might not be reliable, and some preprocessing needs to be done in order to use it. SHAP is simple to use, and can also provide an overall view of the features, and how their values are impacting the model’s predictions. Plus, TreeSHAP leverages the structure of the tree and computes the exact values and not an approximation, however it is less intuitive in terms of explaining it to the business partners.
If you would like to learn more on either of the tools, I encourage you to check out the links embedded throughout the post, as well as the ones below.
4. Further Reading
Here are some additional reading recommendations:
- Explain Your Model with the SHAP Values– a great blog post explaining SHAP with a numerical example among other things.
- A Guide for Making Black Box Models Explainable– provides thorough expiations of interpretable models, LIME, SHAP and much more!
- Local Interpretable Model-Agnostic Explanations (LIME): An Introduction– a blog post written by the authors of LIME.
Special thanks to Or Herman-Saffar and Idan Richman-Goshen for reviewing, and providing valuable feedback on this blog post.