Dynamic Conformal Intervals for any Time Series Model
Apply and dynamically expand an interval using backtesting
Depending on the purpose of generating a forecast, evaluating accurate confidence intervals can be a crucial task. Most classic econometric models, built upon assumptions about distributions of predictions and residuals, have a way to do this built in. When moving to machine learning to do time series, such as with XGBoost or recurrent neural networks, it can be more complicated. A popular technique is conformal intervals — a method of quantifying uncertainty that makes no assumptions about prediction distributions.
Naive Conformal Interval
The simplest implementation of this method is to train a model and hold out a test set. If this test set is at least 20 observations (assuming we want 95% certainty), we can build an interval by placing a plus/minus value on any point prediction that represents the 95th percentile of the test-set residual absolute values. We then refit the model on the entire series and apply this plus/minus to all point predictions over the unknown horizon. This can be thought of as a naive conformal interval.
Scalecast is a forecasting library in Python that works well if you want to transform a series before applying an optimized machine or…