The world’s leading publication for data science, AI, and ML professionals.

Leveraging explainability in real-world ML applications (Part 1)

On the use of explainability to optimize ML models.

Source: https://wepresent.wetransfer.com/story/yes-but-why-yayoi-kusama/
Source: https://wepresent.wetransfer.com/story/yes-but-why-yayoi-kusama/

Why XAI?

Nowadays real-world machine learning applications are ruled by the General Data Protection Regulation (GDPR) law which gives "the right of explanation for all individuals to obtain meaningful explanations of the logic involved" when automated decision making takes place.

Traditional machine learning models such as neural networks are complex and lack transparency and they are considered indeed as Black Box models where data is coming in, and predictions are coming out, without knowing the logic inside that leads to these predictions.

In many safety-critical contexts such as medical or military applications, understanding the internal mechanism and gaining the trustworthiness of the users is a key element in the ML application.

Data bias

In data-driven applications that use a huge amount of data, the need for explanation is also crucial as the data that we collect may contain humans’ biases and prejudices.

One of the examples is COMPASS, a software that is used by U.S. courts to assess the likelihood of a defendant becoming a recidivist. In 2016, the propublica.org (an online newspaper) has shown that according to the profiling scores provided by this software a black who did not re-offend was classified as high risk twice as much as whites who did not re-offend, and white repeat offenders were classified as low risk twice as much as black repeat offenders.

According to Lipton Zachary C. [1], when using ML applications for crime rate predictions to allocate police officers, the racial biases in the training data set may derive the predictions to perpetuate a cycle of incarceration by over-policing some neighborhoods.

Understanding how the predictions of a ML algorithm came up, why in some cases the model can fail and when it can work correctly are essential elements in nowadays ML models for more transparency, trustworthiness and robustness.

XAI aims

  1. providing explanations of predictions
  2. understanding of overall strengths and weaknesses
  3. understanding of how the system will behave in the future.
Source: Broad Agency Announcement Explainable Artificial Intelligence (XAI) DARPA-BAA-16–53
Source: Broad Agency Announcement Explainable Artificial Intelligence (XAI) DARPA-BAA-16–53

How to integrate XAI in ML applications?

To obtain an explainable ML model, it is necessary to take into account the following list of desiderata:

  • Complexity to which extent a human can understand the relationships between the input and the prediction. Measuring the Interpretability is commonly related to the size of the model such as depth of decision tree, tree depth, number of rules, etc.
  • Accuracy to which extent the interpretable ML model accurately predicts unseen instances.
  • Fidelity to which extent the interpretable model is able to mimic the behavior of a black box. It is defined as the accuracy of the interpretable model with respect to predictions of the black-box model.

Global and local explainable model

A global explainable model consists of providing the whole logic and patterns of a model and follows the entire reasoning leading to all the different outcomes. In other words, it explains the conditional interaction between dependent variables (predictions) and independent variables (explanatory variables).

A local explainable model is instead providing an explanation for the prediction of a single instance. It explains the conditional interaction between dependent variables (predictions) and independent variables (explanatory variables) with regard to a single instance.

XAI in real-life applications

Many approaches were proposed to address trustworthiness, fairness, and robustness using XAI models, I cite here some examples:

In [2] the authors presented an explanation model to boost the trustworthiness of the system. The explanation model consists of explaining the predictions of any ML text classifier by learning an interpretable model locally around the prediction and providing insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one.

In [3] the authors proposed an approach that consists of leveraging the paths of a tree-based ensemble classifier to produce recommendations on the transformation of true negative examples into positively predicted examples when tweaking the feature values. This approach was applied in the context of online advertising as it can transform a low-quality ad (true negative instance) into a set of new ‘proposed’ high-quality ads (positive instances) by shifting their position in an ad quality feature space.

In the field of face recognition systems, the authors of [4] presented an approach that provides patterns in failures such as the blurriness of a test image and summarises them with a semantic characterization that humans can understand. These patterns in failure can be used at training time by ML engineers to design better features or collect more focused training data. It can also be used at test time to know when to ignore the output of the system, in turn making it more reliable.

Summary

Most deployed real-world ML applications have been constructed as opaque black boxes. Being able to understand the outcome of the ML model is becoming crucial in many domains. In my next posts, I will describe in more detail the techniques and implemented tools of XAI that allow an ML engineer to integrate easily an XAI model. Stay tuned !!

References

Christoph Molnar. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2018

Defense Advanced Research Projects Agency. Explainable Artificial Intelligence (XAI), 2016

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Turini Franco, Giannotti Fosca and Pedreschi Dino. "A survey of methods for explaining black-box models." ACM computing surveys (CSUR) 51.5 (2018): 1–42.

[1] Lipton Zachary C. "The mythos of model interpretability." Queue 16.3 (2018): 31–57.

[2] Ribeiro Marco Tulio, Sameer Singh, and Carlos Guestrin. " Why should I trust you?" Explaining the predictions of any classifier." Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016.

[3] Tolomei Gabriele, Silvestri Fabrizio, Haines Andrew, and Lalmas Mounia. "Interpretable predictions of tree-based ensembles via actionable feature tweaking." Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017.

[4] Bansal Aayush, Ali Farhadi, and Devi Parikh. "Towards transparent systems: Semantic characterization of failure modes." European Conference on Computer Vision. Springer, Cham, 2014.


Related Articles