Model Interpretability

Explainable Artificial Intelligence (XAI). But, for Whom?

Published in

Towards Data Science

5 min readNov 4, 2021

This blog is a scientific divulgation from “Desiderata for Explainable AI at the European Central Bank” at the ECMLPKDD workshop on Bias (see below citation)

Attempting to explain predictions or build interpretable machine learning models is a hot topic that is quickly expanding. When applying Machine Learning in the decision process in many areas such as medicine, nuclear energy, terrorism, healthcare, or finance, one cannot blindly trust the predictions and leave the algorithm “in the wild,” as that might have disastrous consequences.

The predictive performance of models is evaluated through a wide portfolio of metrics such as accuracy, recall, F1, AUC, RMSE, MAPE… but high predictive performance might not be the only requirement.

There is a huge trend in the field of Explainable AI and Interpretable ML to build trust in the predictions of the model. Many papers, blogs, and software tools present explainability and interpretability in a very defined way, but...

Accountability for whom? Explainability for whom?

I guess that we can all agree on the fact that we don't all need the same type of explanation for how a model behaves. Artificial Intelligence experts such as Andrew NG, Geoffrey Hinton, Ian Goodfellow… won't need the same explanations of why a model is doing what and for which reason, compared to the average data scientist. Moreover, the layperson user probably needs a completely different explanation since they probably have no knowledge of the AI field (nor do they probably wants to).

So, explanations in Artificial Intelligence have different audiences.

Andrew NG and a randomly selected user of the Netflix recommendation system have different needs of understanding “why was this TV series recommended?” A random user might find it sufficient to get suggested a certain action movie because they have been watching action movies every night for the last two weeks, but then Andrew NG (replace name with any other famous AI person) might require a different type of explanation.

Machine Learning explanations are done for a specific audience. Current xAI trends seem to be building explanations that are made for Data Scientists that have a good knowledge of Machine Learning… But should xAI focus on explaining AI to the Data Scientist or to average users?

One typical error in software development is that developers end up developing software for themselves and it turns out to be poorly designed for their target audience [3].

Explainable AI might risk a similar fate as poor software development. Explanations of AI might be developed for data scientists and machine learning researchers themselves and not for the everyday user.

Use case: Lime

One technique that has proved useful is the usage of Local Surrogate models. Probably the most famous is Local Interpretable Model-agnostic Explanations (LIME). The idea is quite simple: the goal is to understand why the machine learning model made an individual prediction; this is called individual explanation. LIME tests what happens to the predictions when you give variations of your data into the machine learning model [5].

In the figure below we can see a local approximation to a certain instance of the well-known wine quality dataset. No matter what the original model is (model agnostic), we can make a local approximation to see which features contribute and for how much.

Lime local explanation for wine quality dataset instance

With this local approximation, we have a linear model that works well just for this data instance. We have perturbed the input data space, defined a local approximation, and fitted a model there.

To whom can we explain this?

Someone can say that this explanation is intuitive for them since it’s built as a linear model estimator with few features. The contribution of each feature is linear. One could even argue that this kind of explanation will be good for a person that has a bit of knowledge in the field, but no previous experience in ML. Also, this can be useful to a Data Scientist that has no time and is going fast through the same interpretations samples for some reason, so a Data Scientist with no time.

But, what are we really doing? Are we using a black-box explainer to predict a black-box algorithm? Only a fraction of the Data Scientists are able to actually understand how a Random Forest or Gradient Boosted Decision Tree behaves, but the fraction of Data Scientists who actually know what LIME does exactly when built out of a Random Forest is even smaller. What is being represented in the LIME local explanation graph? How stable is this explanation? What about its contestability?

Even though the rise of interest in explainable AI is evolving really fast, some questions remain. For whom are we building the explanations? Are we targeting researchers/data scientists? Or are we targeting the general audience?

Citation

This blog is a scientific divulgation from the following published paper(link) at the ECMLPKDD workshop on Bias. Please, cite as:

@inproceedings{MouganDesi,
  author    = {Carlos Mougan Navarro and
               Georgios Kanellos and
               Thomas Gottron},
  title     = {Desiderata for Explainable {AI} in Statistical Production Systems
               of the European Central Bank},
  booktitle = {Machine Learning and Principles and Practice of Knowledge Discovery
               in Databases - International Workshops of {ECML} {PKDD} 2021, Virtual
               Event, September 13-17, 2021, Proceedings, Part {I}},
  series    = {Communications in Computer and Information Science},
  volume    = {1524},
  pages     = {575--590},
  publisher = {Springer},
  year      = {2021},
  url       = {https://doi.org/10.1007/978-3-030-93736-2\_42},
  doi       = {10.1007/978-3-030-93736-2\_42},
}