Araucana XAI: Local Explainability With Decision Trees for Healthcare
Introducing a new model-agnostic, post hoc XAI approach based on CART to provide local explanations improving the transparency of AI-assisted decision making in healthcare
Why did AI get this one wrong?
In the realm of artificial intelligence, there is a growing concern regarding the lack of transparency and understandability of complex AI systems. Recent research has been dedicated to addressing this issue by developing explanatory models that shed light on the inner workings of opaque systems like boosting, bagging, and deep learning techniques.
Local and Global Explainability
Explanatory models can shed light on the behavior of AI systems in two distinct ways:
- Global explainability. Global explainers provide a comprehensive understanding of how the AI classifier behaves as a whole. They aim to uncover overarching patterns, trends, biases, and other characteristics that remain consistent across various inputs and scenarios.
- Local explainability. On the other hand, local explainers focus on providing insights into the decision-making process of the AI system for a single instance. By highlighting the features or inputs that significantly influenced the model’s prediction, a local explainer offers a glimpse into how a specific decision was reached. However, it’s important to note that these explanations may not be applicable to other instances or provide a complete understanding of the model’s overall behavior.
The increasing demand for trustworthy and transparent AI systems is not only fueled by the widespread adoption of complex black box models, known for their accuracy but also for their limited interpretability. It is also motivated by the need to comply with new regulations aimed at safeguarding individuals against the misuse of data and data-driven applications, such as the Artificial Intelligence Act, the General Data Protection Regulation (GDPR), or the U.S. Department of Defense’s Ethical Principles for Artificial Intelligence.
By delving into the inner workings of AI systems and providing explanations for their outputs, researchers strive to demystify the black box, fostering a greater understanding and trust in the technology that is reshaping our world.
Model-agnostic, Post-hoc, Local Explainers
In this blog post, our primary focus will be solely on local explainability, in particular on model-agnostic, post-hoc, local explainers. Model-agnostic explainers can be applied to any machine learning model, regardless of the underlying algorithm or architecture. The term "post-hoc" instead indicates that the explanations are generated after the model has made a prediction for a specific instance. In short, explainers with these properties can analyze any model’s decision-making process for a particular instance, highlighting the features or inputs that had the most significant influence on the prediction, without requiring to modify or retrain the model.
At the time of writing, SHAP [1] and LIME [2] are arguably the two most widely adopted model-agnostic techniques used for explaining the predictions of machine learning models.
- SHAP (SHapley Additive exPlanations) is based on game theory and the concept of Shapley values. It provides explanations by assigning importance scores to each feature in a prediction, considering all possible combinations of features and their contributions to the prediction. SHAP values capture the average marginal contribution of a feature across all possible feature combinations, resulting in a more accurate and consistent explanation.
- LIME (Local Interpretable Model-Agnostic Explanations) approximates the behavior of the underlying model around the prediction of interest by creating a simpler interpretable model, such as a linear model, in the local neighborhood of the instance. LIME explains the model’s prediction by weighting the importance of each feature based on its impact on the local model’s output.
Both SHAP and LIME have their particular strengths and limitations, but one main limitation that is common to both approaches is that they deliver explainability through feature importance and feature ranking. The importance of a feature represents just one aspect of the broader and more complex concept of Explainable Ai. In the clinical domain, for instance, physicians dealing with AI-driven tools often complain about the impossibility of checking and navigating the reasoning process that led the model to land on a specific decision, as they would with a medical guideline.
AraucanaXAI
AraucanaXAI [3] was born to address clinicians’ complaints about traditionally used XAI approaches but can be extended to any other scenario where having decision rules is preferable. The AraucanaXAI framework proposes a novel methodological approach for generating explanations of the predictions of a generic ML model for a single instance using decision trees to provide explanations in the form of a decisional process. Advantages of the proposed XAI approach include improved fidelity to the original model, the ability to deal with non-linear decision boundaries, and native support for both classification and regression problems.
As for SHAP and LIME, also AraucanaXAI is available as a Python package that can be easily installed via PyPI.
GitHub – detsutut/AraucanaXAI: Tree-based local explanations of machine learning model predictions
How does it work?
The algorithm is relatively simple. Given a single instance x:
- Compute D = dist(x,z) for each element z of the training set. The default distance metric is the Gower distance, compatible with mixed-type variables.
- Define subset _Tn as the closest N elements to x (i.e., the neighborhood of x).
- Augment the neighborhood _Tn with SMOTE oversampling (optional). This makes the local region we want to inspect more dense and balanced.
- Re-label the samples of _Tn (or _Tn ∪ S, the samples generated with oversampling) with the class _yhat predicted by the predictive function f of the black-box classifier. Define the explainer set E as _Tn ∪ S ∪ x. Remember that the target of a surrogate model is not to maximize the predictive performance, but to have the same predictive behavior as the original model. That’s why we are interested in _yhat and not y.
- Train the decision tree e on E. Optionally prune it.
- Navigate e from the root node to the leaf node corresponding to x to get the rule set.
Use case example: ALS mortality prediction
Now that we know more about this new XAI approach, when shall we use it? As stated at the beginning of this post, AraucanaXAI is born to address clinicians’ needs, providing explanations in the form of a navigable tree or a set of hierarchical rules that are easy to compare with guidelines and established medical knowledge.
AraucanaXAI has been recently employed to enhance complex prediction models designed to predict mortality for Amyotrophic Lateral Sclerosis (ALS) patients based on observations carried out over a period of 6 months. Predicting ALS progression is a challenging problem requiring complex models and many features, including the administration of questionnaires to stratify the severity of ALS. AraucanaXAI can help clinicians to break down the model’s reasoning into simpler-yet-truthful rules, usually to better understand why the model disagrees with them.
In the example below, for instance, the ground truth says that the ALS patient will die within six months, while a neural network predicts otherwise. The set of rules highlighted by AraucanaXAI can help in understanding the model’s point of view: the onset date happened 3 years before the first ALSFRS-R questionnaire submission (T0), the progression slope is less than 0.35 (i.e., according to the questionnaire, the patient is worsening slowly), the diagnosis date is more than 8 months before T0, and the normalized score for "turning in bed and adjusting bed clothes" is low. Overall, this clinical picture is not that bad and this led the model to think that the patient would still be alive six months after.
Conclusion and Future Work
AraucanaXAI has proven to be a promising approach for XAI for individual patients in healthcare, although the same strategy can be generalized to any other field where breaking down explanations as hierarchical rules constitutes an added value for the decision maker. However, several limitations are still untackled. Firstly, the healthcare field produces an ever-increasing amount of unstructured data, but AraucanaXAI currently works on tabular data. To be employed in the clinical practice, AraucanaXAI should be upgraded to deal also with text and images, which are essential assets for medical institutions. Secondly, the evaluation of what constitutes a "good" explanation for a user cannot be thoroughly assessed without clearly defined metrics (which is currently an acknowledged gap in the XAI literature) and direct involvement of the physician users themselves in a properly designed evaluation study. Such studies constitute future work worth pursuing, with the potential to benefit the explainable AIM community at large. Finally, AraucanaXAI’s way of presenting the generated explanations relies on scikit-learn facilities for decision tree visualization, which is limited and should be improved.
If you are interested in helping with AraucanaXAI, check the GitHub repository and become a contributor! Extension/improvement will be properly credited.
References
[1] S. M. Lundberg and S.-I. Lee – A Unified Approach to Interpreting Model Predictions [2] M. T. Ribeiro, S. Singh, and C. Guestrin – "Why Should I Trust You?": Explaining the Predictions of Any Classifier [3] E. Parimbelli, T.M. Buonocore, G. Nicora, W. Michalowski, S. Wilk, R. Bellazzi – Why did AI get this one wrong? Tree-based explanations of machine learning model predictions [4] T.M. Buonocore, G. Nicora, A. Dagliati, E. Parimbelli— Evaluation of XAI on ALS 6-months mortality prediction
If not mentioned otherwise, images are original contributions of the author.
Please leave your thoughts in the comments section and share if you find this helpful! And if you like what I do, you can now show your support by buying me a coffee for a few extra hours of autonomy ☕
Tommaso Buonocore – Ph.D. Student – Big Data & Biomedical Informatics – ICS Maugeri SpA Società…