Predictive Analytics — Model Predictions And Their Interpretability Challenges

The critical role model interpretability plays in model selection

Photo by Franki Chamaki on Unsplash

Introduction

Humans justify their decisions supported by reasoning. Such reasoning helps discern the decision and repose confidence in it. Sometimes such reasoning logic may not be straightforward but can be laced with anecdotal citations, preconceived intuitions, hypothetical assumptions, and axiomatic intellections. However, as long as such judgment is convincing enough to induce confidence, such decisions are generally acceptable. While the reason may not be straightforward, the extent to which any causality is accounted for, any associations are made with similar events, or the relationships between the underlying factors/circumstances are established would lead to a higher degree of confidence in such decisions. Essentially, this is like trusting the gut to trust the judgment. For example, a patient trusts a treatment plan based on a physician’s prognosis of the ailment, explanation of the diagnosis, experienced-based reasoning, and quoting of comparable example cases.

Similar reasoning is expected when predictive models predict an outcome or make a decision. However, there are critical differences. While several causal, circumstantial, axiomatic, and correlated factors can substantiate human reasoning, the only factors that can ratify predictive outcomes are the example observations, feature characteristics, and the algorithms used to train the predictive models. Such limitations pose challenges to the explicability or interpretability of the predictive models.

Interpretability

Before getting into the meat of the discussion, it is essential to understand what interpretability means in the context of model predictions. There is no industry-agreed definition for model interpretability. However, interpretability can be characterized through the following:

1. Algorithm clarity: This pertains to a comprehensive understanding of how the algorithm works internally, what mathematical intuitions get applied, what computations get executed to process and optimize the inputs, how the processed inputs get mapped to predicted output, and how the algorithm works to ensure accuracy of predictions.
2. Model deconstructability: This implies the ability to break apart the model into granular model units or constructs that explain the partial reasons for the model prediction. For example, a decision node that checks if an applicant’s credit score is greater than a threshold in a decision tree for loan approval. Another example is a hidden node in a neural network that assigns a weight to a specific input variable.
3. Model synthesizability: This implies the ability to walk through every computation using model parameters and the input data manually within a humanly acceptable time frame to derive the model predicted output. Walking through each step in a hierarchical linear model to explain both fixed effects and random effects on the predicted outcome is an example of model synthesizability.
4. Post hoc explainability: This complements all of the above to reinforce the prediction reasoning further. While post hoc explainability does not strictly rely on the internal working of the model algorithm, it complements by providing additional supporting information. Post hoc may employ other utility models to support the reasoning. For an input image, presenting similar images using a k-nearest-neighbor model to justify why a deep-learning image-classification model predicted a specific class is an example of post hoc explainability. Similarly, post hoc may employ model visualizations and other instance-based illustrations to explain the predictions further.

As can be inferred from the above, the computational complexity and the model architecture considerably impact the model interpretability. While computational complexity impacts algorithm clarity and model synthesizability, the complexity of the model architecture impacts model deconstructability. Based on the extent of interpretability, the predictive models can be categorized as black-box or white-box models.

Black-Box vs. White-Box Models

Predictive analytics predominantly solve two broad types of prediction problems — Classification and Regression. In classification, a predictive model would predict a distinct class label as output based on the input variables. In the case of regression, a predictive model would predict a quantitative output such as the price of a product based on the input variables. The primacy on which predictive models are categorized as either black-box or white-box is to do with the model behavior explainability and the interpretability of model results.

Black-Box Models

Black-box predictive models predict outputs given a set of input data. While the relationships between input and output variables are observable from the training dataset, the black-box models either lack the model results interpretability or do not offer a straightforward way to reason why and how the model predicted a particular output. There are two primary reasons for the lack of interpretability. The first reason is that the underlying algorithm is highly sophisticated with complex mathematical computations. Therefore, it is difficult for even seasoned practitioners to comprehend the model working mathematically and translate it into reasoning to support model predictions. The second reason is that while the model’s inner computations are easy to understand, the model setup and architecture become complex at the time of training, where the input data are subjected to several operations and undergo complex transformations. As such, while the underlying superficial working of the algorithm can be explained, it becomes hard to reason a prediction based on all the computations and transformations that input data are subjected to during the processing. Most models with algorithms that leverage complex hyperplanes in multi-dimensional vector spaces, algorithms that leverage complex probability networks, and algorithms that leverage ensembles of sub-models typically fall under the category of black-box models. However, the accuracy of a model increases with the increase in model complexity. Consequently, black-box models generally yield more accurate predictions but at the cost of model interpretability.

White-Box Models

Similar to black-box models, the white-box models predict outputs given a set of inputs. However, white-box models enable interpretability of the relationships between the input and output variables in the training dataset and make it easy to establish similar relationships between the predicted outputs and the given input data. Predictive models that leverage rules-based algorithms, decision trees, decision tables, pattern matching, and algorithms that learn from training instances typically fall under the category of white-box solutions. Such algorithms provide clarity through their inner working and explainability by reasoning how and why the model predicted a particular output. However, the white-box solutions may not provide the level of accuracy that the black-box solutions may provide for complex problem scenarios. Nonetheless, in some cases, the architecture complexity of a white-box model can be upped to increase the model accuracy but at the cost of the model losing the model interpretability. In such scenarios, a white-box model may turn into more of a black-box model. Effectively, one of the key drivers for choosing between the black-box and the white-box models can be a trade-off between accuracy and interpretability.

Functional Types of Models

The functional categorization of the model types also helps to understand how interpretability varies in different model types. The functional types of predictive models can be broadly categorized as follows:

Hyperplane Based Models

These models leverage algorithms that fit hyperplane or hyperplanes in a multi-dimensional vector space using the training data. The fitted hyperplanes separate the classes of the output variables in the high-dimensional vector space. Given input data, the model processes the inputs by applying required transformations. The transformed input vector in the multi-dimensional space can help the model determine which side of the hyperplanes the input vector lies in and select the output class the input vector most likely belongs to. Such models fall under the category of black-box solutions since they implement sophisticated mathematical computations that are hard to explain the reasons for the model predictions. Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Support Vector Machines (SVM), Kernel methods, etc., are examples of hyperplane-based models.

Ensembles with Sub-Models

Ensemble models employ multiple sub-models, in most cases weak sub-models, to improve the accuracy based on the voting or averaging of the predictions made by the sub-models. They use either boosting or bagging techniques to train each of the sub-models. The sub-models learn from the training data either independently or based on the results from the previous sub-model. As such, the ensemble results in a complex decision hierarchy that is hard to explain a predicted output given input data. Therefore, such ensemble models fall under the category of black-box solutions. Random Forest (RF), Extreme Gradient Boosting (XGBoost), Adaptive Boosting (AdaBoost), Bayesian Adaptive Sampling (BAS), etc., are examples of ensemble algorithms.

Probability Network Models

The underlying algorithm builds a complex acyclic directed graph of feature and output nodes in probability network models. Each node maintains the knowledge of all combined conditional probabilities of the prior nodes that connect to it. Essentially, the model picks the output node value as the prediction for which the conditional probability evaluated based on all the previous nodes that the model traverses through is maximum. While the theory of probability such models employ is easy to understand, in real-world cases, the models build a complex graph of nodes with different feature nodes that are computed internally. Therefore it becomes super hard to walk through the model and explain why and how it predicted a particular output. Hence such models fall under the category of black-box solutions. Markov Networks and Bayesian Networks are examples of such models.

Rules-Driven, Patterns-Based, Decision Tree-Like Models

In the case of rules-driven, or patterns-based, or decision tree-like models, the premise of outputs in conjunction with the inputs is well-known and well-understood before developing the predictive models. Such predictive models are developed even after understanding such premise because while the permutations and combinations of such rules and patterns are finite, they are challenging to manage. The model bears the responsibility of such heavy-lifting once the model learns from the training dataset. Because the model makes a prediction based on a particular pattern or a rule-set or by traversing a decision tree, it is easy to interpret the prediction based on the intermediate decisions that the model had to make considering each input variable. Therefore such models typically fall under the category of white-box solution. However, if the rule-sets become too hierarchical or if the decision tree becomes too deep and nested, the interpretability may take a hit because the resulting convoluted decision hierarchy may be too challenging to explain. In such cases, the model turns into a black-box solution.

Linear Models

The linear models establish linear relationships between the input and output variables. The strength of such relationships is estimated in the form of linear coefficients that act as the model parameters. The quantified estimates of the parameters help determine the importance of each input feature relative to the output variable and help explain why and how the model predicted an output based on the given input data. As such, the linear models typically fall under the category of white-box solutions. However, if the linear model has high dimensional input space or if several computationally engineered variables or dummy variables are appended at the model development time, the linear model may lose its interpretability. Linear Regression and Logistic Regression are examples of such linear models.

Instance-Based Models

In the case of instance-based models, the algorithm makes a prediction based on localized approximations between the given input data and the past instances from the training data that exhibit similar characteristics based on a specified target function and then employing a strategy to determine the realm of the locality to select the instances. Once the local instances that are functionally similar to the given input are chosen, the algorithm picks the prediction based on voting or averaging. Because the prediction is based on local approximations using similar instances, the model exhibits a high degree of interpretability. Effectively, the characteristics of the similar instances chosen locally can explain the model’s predictive behavior. As such, instance-based models typically fall under the category of white-box solutions. K-Nearest Neighbor (kNN), Self-Organizing Maps (SOM), etc., are examples of instance-based models.

Model Selection

The determination of what model solution is most appropriate depends on several factors that include factors outside of the realm of the underlying algorithm subject matter and technicalities. For example, black-box solutions would be right where the model predictions are expected to be accurate but are understood to be beyond human discernability. In such cases, the model performs complex operations to establish relationships between the input and output variables that are incomprehensible to human understanding. For example, the prediction of traffic volumes at critical road intersections in connected vehicles use case. In such a case, the model may need to use data from roadside sensors, data about nearby accidents or road repairs, traffic peak hours, etc., to predict traffic volumes accurately. The use of complex algorithms such as Artificial Neural Networks (ANN) or Support Vector Regressions (SVR), or Extreme Gradient Boosting (XGBoost) is warranted because:

1. The solution intends to predict traffic volume to prevent any road accidents.
2. There are no preset patterns to establish rules to prescribe what combinations of different inputs would determine the estimate of the traffic volume. The number of permutations and combinations would proliferate beyond what humans can comprehend.
3. The model predictions can be easily correlated to those that humans could predict, and therefore the model results can be validated.

Likewise, there are scenarios where it becomes mandatory to employ white-box solutions. For example, the cases where reasoning and causality take precedence because the underlying problem domain mandates such reasoning for predictions, or compliance regulations enforce such a reasoning requirement. For example, the United States Equal Credit Opportunity Act deems it illegal to deny a bank loan to an applicant where the reasons are vague or unjustifiable. As such, the financial institution may have to employ a white-box model that can classify and, at the same time, provide a straightforward way to explain the satisfactory reasons behind the predicted decisions. Similarly, in healthcare and medicine, where decisions need justifiable reasoning and causality attribution, it may become critical to employ white-box solutions for making medicinal predictions. In all such scenarios, it is no longer the pure accuracy of the prediction that matters, but interpretability becomes critical. Therefore the selected model should be based on the compromises between the two concerns. In some other scenarios, the model simplicity may drive the selection of more straightforward white-box solutions.

Nevertheless, some situations can leverage a combination of black-box and white-box models to develop predictive analytics solutions. In such cases, the black-box model is employed to make primary predictions, but post-hoc white-box models complement the predictions’ interpretability. For example, the interpretability of a CNN model for image classification will be hard because CNN employs complex convolutional operations, image filters, and transformations that lack discernability through human explanations. However, a separate, simpler white-box model such as a decision tree or a clustering model can be developed using the complex features extracted for each training image by the CNN. Such a white-box model can then match similar images from the training data to enable post-hoc reasoning and interpretability.

Conclusion

Human understandable reasoning is expected when predictive models predict an outcome or make a decision. The only constituents that can substantiate the explainability of predicted results are the example observations, feature characteristics, and the algorithms used to train the predictive models. Such limitations pose challenges to the interpretability of the predictive models. While there is no standard definition for model interpretability, the underlying algorithm clarity, the model deconstructability, the model synthesizability, and the post hoc explainability characterize the overall model interpretability.

Based on the extent of interpretability, the predictive models can be categorized as black-box or white-box models. Black-box models lack the model results interpretability or a straightforward way to reason why and how the model predicted a particular output. In contrast, white-box models enable the interpretability of the relationships between the input and output variables in the training dataset and make it easy to establish similar relationships between the predicted outputs and the given input data. Models with algorithms that leverage complex hyperplanes in multi-dimensional vector spaces, algorithms that leverage complex probability networks, and algorithms that leverage ensembles of sub-models typically fall under the category of black-box models. Models that leverage rules-based algorithms, decision trees, decision tables, pattern matching, and algorithms that learn from training instances typically fall under the category of white-box solutions.

The determination of what model solution is most appropriate depends on several factors that include factors outside of the realm of the underlying algorithm subject matter and technicalities. Black-box solutions would be right where the model predictions are expected to be accurate but are understood to be beyond human discernability. Use cases where reasoning and causality take precedence because the underlying problem domain mandates such reasoning for predictions, or compliance regulations enforce such a reasoning requirement warranty using white-box models. Some use cases leverage a combination of black-box and white-box models to develop predictive analytics solutions. In such cases, the black-box model is typically employed to make primary predictions with greater accuracy, and white-box models complement the predictions’ post hoc interpretability.

Since the accuracy of a model increases with the increase in model complexity, black-box models generally yield more accurate predictions but at the cost of model interpretability. Effectively, one of the key drivers for choosing between the black-box and the white-box models can be a trade-off between accuracy and interpretability. As such, it is imperative that the data scientists and machine learning experts first understand the problem domain, its objectives, and the underlying compliance and reasoning requirements before determining the choice of model algorithms to develop predictive analytics solutions. Model interpretability becomes crucial to meet all such mandatory requirements while balancing the model accuracy.