Dynamic Conformal Intervals for any Time Series Model

Apply and dynamically expand an interval using backtesting

Michael Keith
Towards Data Science

--

Photo by Léonard Cotte on Unsplash

Depending on the purpose of generating a forecast, evaluating accurate confidence intervals can be a crucial task. Most classic econometric models, built upon assumptions about distributions of predictions and residuals, have a way to do this built in. When moving to machine learning to do time series, such as with XGBoost or recurrent neural networks, it can be more complicated. A popular technique is conformal intervals — a method of quantifying uncertainty that makes no assumptions about prediction distributions.

Naive Conformal Interval

The simplest implementation of this method is to train a model and hold out a test set. If this test set is at least 20 observations (assuming we want 95% certainty), we can build an interval by placing a plus/minus value on any point prediction that represents the 95th percentile of the test-set residual absolute values. We then refit the model on the entire series and apply this plus/minus to all point predictions over the unknown horizon. This can be thought of as a naive conformal interval.

Scalecast is a forecasting library in Python that works well if you want to transform a series before applying an optimized machine or…

--

--

Responses (3)