Open in app

Sign up

Sign in

Write

Sign up

Sign in

All You Can Game Buffet: Model Risks in the Climate-Related Physical Risks

A checklist of model risks associated with CRPRs

Michio Suginoo
Towards Data Science

--

Photo by Mike Newbry on Unsplash

In the age of Climate Change, Data Scientists can play a significant role in pursuing sound data-driven decision makings to promote sustainability: to robustly model the climate-related risks (CRRs) and provide useful information to decision makers to alleviate negative consequences arising from the risks. Therefore, CRRs are highly relevant to Data Scientists. There are a wide range of risks associated with modeling CRRs. In this context, data science professionals involved in modelling climate-related risks would need to have a solid understanding regarding those risks. In this light, I decided to compile my past research works (e.g. Suginoo, 2022) on the topic to present my personal view at TDS. I hope that this post will give data science professionals a basis for a checklist in addressing model risks associated with CRRs.

INTRODUCTION

Climate-related Risks (CRRs) are path-dependent and highly uncertain. CRRs arise from a highly complex climate system, and thus, are difficult to model.

Model Risks & Tail Risk

Typically, there are two types of risk associated with model uses: model risk and tail risk.

  • Model risk is “the risk of a valuation error from improperly using a model. This risk arises when an organization uses the wrong model or uses the right model incorrectly.” (Chance & Edleson, 2019, p. 21)
  • Tail risk is present when we find more events in the tail of the actual distribution than would be expected by the probability models in use. (Chance & Edleson, 2019, p. 21)

To a great part, the financial crisis of 2007/8 might illustrate an example of these two risks. While extensively using advanced portfolio risk models — such as Value at Risk — the financial system expanded the liabilities (on and off the book) beyond their capacity. Those sophisticated models rather encouraged the financial industry to justify excessive risk takings. The use of those models ended up accounting for an illusion of knowledge and an illusion of control. It fed the causes of a systemic financial crisis and miserably failed to protect the system from the fat tail event. (Nocera, 2009)

CRRs, if not alleviated, could manifest progressively increasing correlations and unprecedented non-linear developments among risk variables. (The Climate Change Working Party, 2020, p 9) They could drive an irreversible systemic paradigm shift in the climate system’s equilibrium. (Schneider S. H., u.d., Abstract)

CRRs could pose detrimental fat tail risk to our human life. Given the historical lesson of the financial crisis, it would be imperative for data science professionals to address model risks and tail risk (Schneider, 2003, p5) and robustly model CRRs from a data-driven risk management perspective.

The Physical Risk & The Transition Risk of Climate-related Risks

Taskforce of Climate-related Financial Disclosure (TCFD) — an industry-led task force whose mission is to develop guidelines for voluntary climate-related financial disclosures — divides CRRs into two classes of risks: the transition risks and the physical risks.

  1. Transition Risks might arise during the transition to a lower-carbon economy. And these are risks associated with “policy, legal, technology, and market changes to address mitigation and adaptation requirements related to climate change”.
  2. Physical Risks can be categorized into acute (event-driven) or chronic risks.
  • Acute risks are “event-driven, including increased severity of extreme weather events, such as cyclones, hurricanes, or floods. such as hostile climate.”
  • Chronic risks “refer to longer-term shifts in climate patterns”: e.g., an elevated sea level or chronic heat wave arising from sustained higher temperatures.

(TCFD, 2017, pp. 5–6)

Below, based on TCFD’s classification, this post will focus on the case of the physical risks of CRRs — call it the climate-related physical risks, CRPRs — and explore potential risks associated with modelling CRPRs. And, I hope that this post will give data science professionals a basis of check-list to address a wide range of model risks involved in CRRs.

DISCUSSION

In the past, property and casualty insurance industry — property & casualty insurers and re-insurers— has developed their own catastrophe/hazard models to assess insurance risks associated with catastrophe risks. In this backdrop, it is common to apply their framework to model the climate-related physical risks. (UNEP Finance Initiative, 2019, p. 8)

First, the section A explores potential sources of uncertainties involved in the property & casualty insurers’ catastrophe models. Then, the section B will explore potential risks present in applying the framework of the property & casualty insurers’ catastrophe models to alleviate the climate-related physical risks.

A. Sources of Uncertainties involved in Insurer’s Catastrophe Models

a) 2 Systematic Sources of Uncertainties

In addition to the model risk and tail risk, a paper published by Lloyd’s of London is concerned with the following two systematic sources of uncertainties with the climate-related physical risks. (Toumi & Restell, 2014)

  • the internal variability in the climate system itself and
  • the influence of anthropogenically-induced change.

Apart from anthropogenically-induced change, such as exponential rise in GHG emissions induced by human activities, the climate system itself has already internal variabilities within. We often observe droughts taking places in some areas while flooding taking places in others at the same time. The inherent internal variability of the climate system, call it natural variability, makes it difficult for scientist to generalize the consequences of the climate-related risks with or without anthropogenically-induced change.

b) Trends

In addition, natural variability and anthropogenically-induced changes combined together are demonstrating new emerging trends. (Toumi & Restell, 2014, p. 32) The Lloyds’ paper articulates the necessity of incorporating observed trends into the model as follows:

“Where a catastrophe model does quantify losses for a region/peril, the process is complex and depends on many assumptions which naturally result in a degree of uncertainty around that loss. This uncertainty increases for more extreme events where there is little empirical experience and instances where exposure data imported into the catastrophe model by the client is of poor quality. It is paramount that the limitations of the model and the uncertainty inherent in its outputs are conveyed effectively during the decision making process. In order for catastrophe models to assist in the forecasting of risk exposure, they must incorporate observed trends.” (Toumi & Restell, 2014, p. 9)

Altogether, the mixture of these two variabilities would make it difficult for scientists to model the impacts of the CRPRs.

In other words, no single model can remove these two inherent uncertainties from the climate system today, simply because they are systematic. Thus, any output from the model would convey these uncertainties. Simply put, we cannot make any precise forecast about CRPRs using models in one single value, say point-estimator. Any single point-estimator would be deceptive, when we deal with CRRs.

All that said, we can still explore a range of potential hypothetical scenarios to prepare for the future uncertainties. We need to capture the risk of scenario paths with probabilistic range (e.g. confidence intervals) to test the resiliency of the current strategy in place.

c) Vendor-specific Biases

Furthermore, the Lloyds’ paper includes another source of uncertainties which could arise when we use a vendor model: vendor-specific biases. Simply put, any vendor model is predicated on a certain set of assumptions to enable itself to analyze the complexity of CRPRs. Those assumptions inevitably are the source of model biases and set the scope and limitations of the model in use.

Overall, in the Lloyds’ paper, Toumi & Restell identifies a handful of sources of uncertainties in assessing the property & casualty insurers’ catastrophe/hazard models (Toumi & Restell, 2014, p. 9):

  1. model risk and tail risk
  2. systematic sources of uncertainties of CRPRs:
  • natural/internal variability of the climate system
  • the influence of anthropogenically-induced change

3. vendor-specific biases

So far, we discussed the sources of uncertainties embedded in the property & casualty insurers’ catastrophe models within the context of the property & casualty insurers’ risk management framework.

Inevitably, the property & casualty insurers’ model framework inherently suffers from their business biases.

In other words, there are some potential limitations in applying the framework of their catastrophe/hazard models to alleviate CRPRs in a broader context.

B. Potential risks in using Insurer’s Catastrophe Model to model Climate-related Physical Risks

The property & casualty insurers’ business biases account for profile mismatches with CRPRs over the following three factors: namely,

  • Risk-profile mismatch,
  • Time-horizon mismatch, and
  • Scenario base mismatch.

We will cover these mismatches one by one in this section.

a) Risk-profile Mismatch:

Generally speaking, conventional insurance business operates on the principle of the law of large numbers. (Smith & Kane, 1994) Conceptually, assuming diversifiable non-systemic risk — which is expected to materialize at a stable known probability (stationarity paradigm) — a property & casualty insurer can pool the risk and diversify it among a large number of the insureds.

Here is a simplified explanation how the law of large number works in the conventional insurance business.

  • First, the property & casualty insurer makes a sufficiently large target sales volume scenario for an insurance product; then applies the expected probability of the insurance claim frequency to the target insureds in order to estimate the expected total insurance losses (claims) among the hypothetical target insureds.
  • Thereafter, on the top of the expected total insurance losses (claims), they mount the total operating costs and the profit margin to estimate the total revenue required for the particular insurance product.
  • Finally, they divide the total required revenue by the target insureds to price the insurance premium for the insurance product.

This is a simplified version of the insurance premium pricing (actuarial pricing).

Simply put, the property & casualty insurers make profit by diversifying non-systemic risks among the given insureds. Underneath of this mechanism there is a critical assumption of the stationarity. Stationarity is a state in which statistical properties, such as mean and variance, would not change over time. The property & casualty insurers can expect profit because they assume that the insurance risks are in the stationarity paradigm.

In contrast, CRPRs are manifesting an irreversible departure from the past equilibrium. Since we are in an irreversible transient state, the statistical properties of CRPRs would change. Thus, it would be wrong to assess CRPRs in a stationarity paradigm to begin with. The next block quote articulates the importance in distinguishing between stationarity and non-stationarity behaviors in dealing with Climate System.

In addition, since CRPRs are intensifying across all geographical locations, it would be increasingly systemic, thus, difficult to diversify their CRPRs’ coverage geographically toward the future.

To cope with this particular limitation of conventional insurance risk management frameworks, there emerged a new class of products called Alternative Risk Transfer (ART). The idea of ART is basically a zero-sum game by design. Catastrophe Bonds (CAT bonds) is an example of ART. The property & casualty insurers may sell CAT bonds to pass the systemic risks to buyers in exchange for a series of coupon-like instalment payments. Inevitably, in the age of climate change, as the expectation of CRPRs rises, CAT Bonds would become progressively difficult to market.

Overall, those non-systemic and stationarity assumptions embedded in the conventional insurance risk management framework would not be compatible with the systemic and non-stationarity risk profile of CRPRs. In this sense, naïvely copying the frameworks of the property & casualty insurers’ catastrophe/hazard risk management framework to assess CRPRs might suffer from risk-profile mismatches: systemic vs non-systemic; stationarity vs non-stationarity.

b) Time Horizon Mismatch

CRPRs are unprecedented and uncertain both in development and in timing. Therefore, they are multi-time horizon: short-, medium-, and long-term horizons.

On the contrary, in many cases, the terms of property insurance programs are typically one year horizon. Inevitably, this business practice has shaped their tendency to set the time-horizon of their catastrophe/hazard risk management framework to be short.

In the following excerpt from a climate-related financial disclosure of AIG, one of major general insurers, we can see an example of the risk management framework of the general insurers:

“A meaningful proportion of our general insurance policies are renewed on an annual basis, providing us the opportunity to re-underwrite and re-price the risk regularly. Medium- and-long-term impacts are considered in strategy setting and asset liability management decisions in both the General Insurance and Life and Retirement businesses. Fundamental trends and significant changes over longer horizons are more challenging, as precise forecasts are difficult to make.” (AIG2, 2020, pp. 8–9)

Their short-term biases of the property & casualty insurers’ business would not match with the multi-time horizon behavior of CRPRs. It might well account for the source of uncertainty in simply copying the property & casualty insurers catastrophe risk management frameworks for the purpose of alleviating CRPRs. In other words, when applying the property & casualty insurers’ catastrophe risk model to manage CRPRs, the users need to adjust the time horizons appropriately.

Furthermore, there is an additional mismatch arising from their short-term bias, as discussed below.

c) Scenario Base Mismatch (Dataset Bias)

Based on the assumption that the short-term future can be projected based on historical data, the insurance industry uses historical data and makes a set of adjustments to assess the catastrophe risk within a short-term horizon.

Tautologically, the past has no information about the unprecedented future that CRPRs would manifest.

“The historical simulation method has the advantage of incorporating events that actually occurred and does not require the specification of a distribution or the estimation of parameters, but it is only useful to the extent that the future resembles the past.” (Chance & McCarthy Beck, 2021, p. 49)

Any model that employs historical datasets need to be scrutinized at the users’ end for the appropriateness of the embedded assumptions for use.

Especially, naïvely applying data-driven machine learning algorithms, such as Deep Learning, over the historical data can deceptively overfit the model to the past pattern and account for the model instability (variance). As result, we might end up underestimating the future risk. Although being good at discovering the historical pattern out of the dataset, they are not designed to anticipate unprecedented non-stationarity changes in the future pattern among the risk variables.

CRPRs are path-dependent and extremely uncertain. Thus, there are many potential future paths for CRPRs in front of us. Certainly, no one knows which path will unfold. In this context, TCFD recommends exploratory scenario analysis to forecast CRPRs since we can incorporate into the model multiple hypothetical scenario paths across multi-time-horizons.

Nevertheless, scenario analysis suffers from its own inherent limitation by design. It relies on subjective hypothetical scenarios. Recently, Ph.D. Tony Hughes made a LinkedIn post to articulate the inherent uncertainties involved in scenario analysis: there is no guarantee for the validity of the assumptions; misspecifications of the statistical models can lead to misleading outcomes. (Hughes, 2022)

A screenshot: an excerpt from Ph.D. Tony Hughes’ LinkedIn post on 20th of October 2022

CONCLUSION

We discussed a handful sources of uncertainties involved in the property & casualty insurers’ catastrophe model.

  1. Model risk and tail risk
  2. Systematic sources of uncertainties of CRPRs:
  • Natural/internal variability of the climate system
  • The influence of anthropogenically-induced change

3. Vendor-specific risks

In other words, it is imperative to do due diligence to go over the following check list in the selection of the model to assess the climate-related physical risks.

  • Whether tail risks are incorporated into the vendor model in use: how the vendor model incorporates intensified complexity and uncertainty in modeling an unprecedented extreme event
  • How the model enables the users to incorporate observed trends in assessing CRPRs
  • How the model specifications convey the limitations of the model and the uncertainty inherent in its outputs for decision-making process.

In addition, I articulated my personal perspective that the common practice of applying the property & casualty insurers’ catastrophe model for assessing CRPRs could suffer from model risks due to the following 3 types of mismatches:

  • Risk Profile Mismatch: systemic vs non-systemic risk; non-stationarity vs stationarity behaviors
  • Time Horizon Mismatch: multi-horizon vs short term
  • Scenario Base Mismatch (Database Bias): future scenarios vs historical data

All that said, I have no intention to denounce the property & casualty insurers’ catastrophe/hazard risk models. Instead, I rather want to call for users’ attention to the potential risks associated with applying the property & casualty insurers’ risk management framework to alleviating CRPRs in a broader sense.

Overall, if a user chooses to use the property & casualty insurers’ models, the user could customize the property & casualty insurers’ risk management frameworks by incorporating the following three characteristics of CRPRs into their analysis for their use in CRPRs management:

  • Non-diversifiable systemic and non-stationarity risk profile
  • Multi-time horizon
  • Potential multiple future scenarios

Probably this is not a comprehensive list to cover all the risks associated with the use of the property & casualty insurers’ catastrophe models in alleviating CRPRs in a broader context.

Apart from all those risks, potentially, some politically charged end-users could game a subjective/biased scenario to generate an underestimated prediction in order to manipulate the public opinion of CRPRs’ impact. Scenario analysis can serve all you can game type — like all you can eat buffet — of abuse of models.

Whatever models we use, we are all the prisoner of our corrupted and/or precarious human nature. No model would liberate us from our self-delusions and political manipulation.

Overall, data science professionals need to raise self-awareness on these human defects and the vulnerability of any model of being gamed/abused and well communicate with the audiences regarding the limited scope of whatever model in use: especially its assumptions and the implications of the model’s outputs. (The Climate Change Working Party, 2020, p. 10)

At last, I would like to end the article with an insightful remark made by a British Statistician, George Box:

“All models are wrong, but some are useful” (Wikipedia, 2022)

I hope that this post will give data science professionals a basis of check-list in addressing model risks associated with CRRs.

Thanks for reading.

REFERENCE

Climate Change
Systemic Risk
Data Science
Machine Learning
Data For Change

--

--

Michio Suginoo
Towards Data Science
Follow

Written by Michio Suginoo

99 Followers
Writer for

CFA, Data Science, Innovation, Paradigm Shift, Paradox Hunting, Teleological Pursuit

Follow

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams