The world’s leading publication for data science, AI, and ML professionals.

Explaining a CNN generated soil map with SHAP

Using SHAP to corroborate that the digital soil mapping CNN is capturing sensible relationships.

This is the fourth article of a Series that I am devoting to the use of Deep Learning in Soil Science. This is an ongoing series and so far it also includes:

Deep learning and Soil Science – Part 1

Deep learning and Soil Science – Part 2

Deep learning and Soil Science – Part 3


In one of my previous articles, I explained how I use spatial models to generate maps. I also introduced the use of a multi-task convolutional neural network (CNN) to better represent the complexity of the soil formation process. The CNN uses a window of pixels around a punctual soil observation as input instead of the single pixel intercepting its location, in order to incorporate spatial context. In addition, thanks to its multi-task design, the model is capable of predicting multiple soil properties simultaneously, taking into account the interaction between them. The introduction of these two novel features yielded an error decrease of around 30% compared with a more conventional ML model (Cubist tree model).

You are probably aware of how Deep Learning have shown excellent performance in many fields but that at the same time they have raised concerns about their interpretability and potential biases, especially since these type of algorithms are starting to be applied to assist decision-making in domains with high human impact.

To address the interpretability issue, in this post I show some results of my latest research where I apply SHAP to interpret a multi-talk CNN for digital soil mapping. The main objective was to corroborate that the model is capturing sensible relationships between the target soil property and the covariates used to train the model.

Game theory and SHAP values

From a game theory perspective, modelling something can be seen as a collaborative game where the predictors (agents or participants) strategically interact to achieve the goal of predicting an output. As a result, each agent receives a payout proportionate to their contribution.

Shap is a method to approximate the marginal contributions of each predictor. For details on how these values are estimated, you can read the original paper by Lundberg and Lee (2017), my publication, or an intuitive explanation in this article by Samuele Mazzanti.

For the analysis, I used the shap python library by Scott Lundberg, one of the creators of the method.

Model description

The mode in question is a multi-task CNN, trained to simultaneously predict soil organic carbon (SOC) content of Chile at five depth intervals (0–5, 5–15, 15–30, 30–60 and 60–100 cm). To train the CNN, I used 3D arrays with shape (29, 29, 5), which represent a 29×29 pixels window around each observation, for the five covariates used to explain the spatial distribution of SOC. The five covariates included in this simple model are: elevation, slope, topographic wetness index (TWI), long term mean annual temperature (MAT) and total annual precipitation (TAP).

Figure 1. Architecture of the multi-task network. "Shared layers" represent the layers shared by all the depth ranges. Each branch, one per depth range, first flattens the information to a 1D array, followed by a series of 2 fully-connected layer and a fully-connected layer of size=1, which corresponds to the final prediction.
Figure 1. Architecture of the multi-task network. "Shared layers" represent the layers shared by all the depth ranges. Each branch, one per depth range, first flattens the information to a 1D array, followed by a series of 2 fully-connected layer and a fully-connected layer of size=1, which corresponds to the final prediction.

Results

Local and global explanations

If you are familiar with SHAP, probably you have seen some of the following figures. First, we can obtain a local explanation (a single prediction) with the contribution of each covariate. In the figure below we can see the force plots for two different observations: the top panel corresponds to a mountainous area where the elevation and slope have a negative contribution (they promote erosion). The bottom panel corresponds to a valley where the opposite is true. Both observations are located in the south of Chile, where colder and wetter conditions promote the accumulation of SOC compared to the central and northernmost regions, hence the positive contribution of temperature and precipitation.

Figure 2. Force plots of two different samples. The figures depict the positive (green) and negative (purple) aggregated contributions that deviate the SOC content from the expected base SOC value (the mean of the predictions for the complete dataset).
Figure 2. Force plots of two different samples. The figures depict the positive (green) and negative (purple) aggregated contributions that deviate the SOC content from the expected base SOC value (the mean of the predictions for the complete dataset).

With SHAP is also possible to obtain a global explanation of the model, considering the contribution of the covariates over all the samples. The next figure shows the relative importance of each of the covariates, where climate has a greater influence (a broader range of SHAP values) over topography related covariates. This is a national SOC model and Chile has a strong north-south climatic gradient, so it is not surprising that TAP and MAT have the greatest influence. At a more local level, topography starts playing an important role. Another thing that the figure below shows us is that SOC has a positive correlation with precipitation and a negative correlation with temperature, which is also an expected behaviour.

Figure 3. SHAP values for each covariate and soil depth interval for the CNN model. TAP: total annual precipitation; MAT: mean annual temperature; TWI: topographic wetness index.
Figure 3. SHAP values for each covariate and soil depth interval for the CNN model. TAP: total annual precipitation; MAT: mean annual temperature; TWI: topographic wetness index.

Interactions

It was also possible to further inspect the effect of particular covariates and evaluate if the model captured interactions between them. For instance, the positive contribution of temperature peaked in the range between 4 and 8 ºC with a negative contribution above 12 ºC. Given the climatic conditions in the country, the decrease in MAT contribution is correlated with the decrease of precipitation, which is accentuated in areas with high temperatures (Fig. 4a) where carbon inputs are low. The contribution of precipitation (Fig. 4b) showed a defined trend, with a mostly constant negative response up to around 1,000 mm yr ⁻¹ where it starts increasing, becoming positive around 1,400 mm yr ⁻¹. The model also captured an inverse relationship between SOC content and elevation (Fig. 4c) which higher values are usually associated with steeper terrain, given the topography of the country. In general, this effect was exacerbated in areas with TAP higher than 400 mm yr ⁻¹.

For both, temperature and precipitation, the threshold values where their contributions turn positive (around 12 ºC and 1,400 mm yr ⁻¹, respectively) coincide with a significant area within the country (around 38º latitude S.), where Andisols ("volcanic" soils) become more prevalent and there is a change from a xeric to an udic soil moisture regime (it becomes more humid), associated with a sharp increase in the content of SOC. In terms of the TAP threshold that seems to change the behaviour of the contribution of the elevation (around 400 mm yr ⁻¹), it roughly corresponds to the transition between the arid and semi-arid zones of the country (becomes more humid), where the erosion processes by water start becoming more important.

Figure 4. Dependency plots between SHAP values and selected covariates. a) Between SHAP values and mean annual temperature (MAT), showing the interaction with total annual precipitation (TAP; colour scale); b) between SHAP values and total annual precipitation , showing the interaction with mean annual temperature (colour scale); and c) Between SHAP values and elevation, showing the interaction with total annual precipitation (colour scale).
Figure 4. Dependency plots between SHAP values and selected covariates. a) Between SHAP values and mean annual temperature (MAT), showing the interaction with total annual precipitation (TAP; colour scale); b) between SHAP values and total annual precipitation , showing the interaction with mean annual temperature (colour scale); and c) Between SHAP values and elevation, showing the interaction with total annual precipitation (colour scale).

Spatial explanations

Since we are talking about a spatial model that incorporates context around each sample, we can also evaluate the contribution spatially. The figure below shows the same two locations from Fig. 2, but this time disaggregating the contribution per pixel. The interpretation is the same but, in theory, we should be able to identify specific features of the landscape that are more influential.

Figure 5. Spatial distribution of SHAP values for each covariate in two different samples (the same samples from Fig. 2).
Figure 5. Spatial distribution of SHAP values for each covariate in two different samples (the same samples from Fig. 2).

Perhaps the most interesting output for me (that I mostly work with maps) is that it is possible to aggregate the local spatial contributions to show a more general spatial interpretation and further corroborate the previous explanations. The figure below shows a map of SHAP values where it is possible to distinguish the valley close to the coast, which has lower TAP and slope, and higher MAT compared with the surrounding areas. Those conditions contribute negatively to the model output, compared to the expected mean output. Of course, even when those trends make sense in terms of SOC processes, they are probably interacting with the effect of human activity (agriculture) in the valley, which is not accounted for in this model.

Figure 6. Spatial distribution of SHAP values for each covariate for the CNN model. The value of each pixel is an average of all the instances where that pixel is used as context (up to 841 times for a 29x29 context window). TAP: total annual precipitation; MAT: mean annual temperature; TWI: topographic wetness index.
Figure 6. Spatial distribution of SHAP values for each covariate for the CNN model. The value of each pixel is an average of all the instances where that pixel is used as context (up to 841 times for a 29×29 context window). TAP: total annual precipitation; MAT: mean annual temperature; TWI: topographic wetness index.

Final words

The results of this study show that it is possible to work towards interpretable deep learning models in digital soil mapping and that a complex model, generally perceived as a black box, can be inspected using SHAP values. This is not only necessary to corroborate that the model was trained properly and that it capturing sensible relationships between the target property and the predictors, but it is also a fundamental part of the process leading to knowledge discovery.

Citation

More details about this work can be found in the corresponding paper.

Padarian, J., McBratney, A. B., and Minasny, B. 2020. Game theory interpretation of digital soil mapping convolutional neural networks, SOIL Discuss, https://doi.org/10.5194/soil-2020-17.

Note 04/04/2020: The paper has not been accepted yet and it is under public review and discussion. You are welcome to participate in this process.

References

  • Lundberg, S.M. and Lee, S.I., 2017. A unified approach to interpreting model predictions. In: Advances in neural information processing systems 30. pp. 4765–4774.

Related Articles