Forecasting Fundamentals You Should Know Before Building Predictive Models
To forecast, or not to forecast, that is the question.
The history of human civilization is entwined with the history of methods we have tried with forecasting. While our ancestors observed the sky to forecast the weather, Data Scientists develop and train machine learning models to predict sales, risks, events, trends, etc. Accurate forecasting is a critical organizational capability, and businesses that do it well would have a big advantage to survive.
What should be forecast
In the early stages of a forecasting project, decisions need to be made about what should be forecast, when something can be forecast accurately, and when forecasts will be no better than tossing a coin.
For example, if forecasts are required for the Real Estate Market, it is necessary to ask whether forecasts are needed for:
- Any type of property, or for Detached/Semi-Detached/Townhouse/Condo separately?
- The average sales price for properties grouped by region, or only for total sales?
- Weekly data, monthly data or annual data?
The key that needs to pay attention is that not all forecasts are better than a coin toss. The predictability of an event or a value depends on several factors including:
- Do we discover and understand most of the factors that contribute to the result?
- Are there “big” data for those factors?
- Whether the prediction can affect the thing we try to predict?
For example, 24-hour weather forecast can be highly accurate because all three conditions are usually satisfied. On the other hand, it is difficult to predict the probability of next raindrop falling into your eyes, the concurrency exchange rate will rise or fall tomorrow or Bitcoin price for next week.
Forecasting types in terms of time horizon and data availability
Will forecasts be required for a few minutes in advance, for 6 months, or for ten years? There are 3 types of forecasting in terms of time horizon:
- Short-term forecasts: a normal range between one and three months
- Medium-term forecasts: the time period is normally one year
- Long-term forecasts: predict results over periods greater than two years
In most forecasting situations, the uncertainty associated with the thing we are forecasting will decrease as the event approaches. In other words, the closer ahead we forecast, the more accurate we are.
Forecasting can be categorized into 2 types in terms of data availability.
- Qualitative Forecasts: If there are no data available, or if the data available are not relevant to the forecasts.
- Quantitative Forecasts (Time Series Forecasting): If numerical information about the past is available; and the past patterns will continue into the future.
Data for Quantitative Forecasts (Time Series Forecasting) are often observed at regular intervals of time, e.g., hourly, daily, weekly, monthly, quarterly, annually. The goal is to estimate how the sequence of historical observations will continue into the future.
Forecasting Methods
The choice of method depends on what data are available and the predictability of the event or value to be forecast.
Judgmental Forecasting is the only option for Qualitative Forecasts due to the lack of historical data. For example, forecast the effect of a new policy, a new product, or a new competitor.
The simplest Time Series Forecasting methods only use historical values on the variable to be forecast and exclude the factors that could affect its behavior such as competitor activities, changes in environment or economic conditions, and so on. The following image shows house sales prediction for 2017 by using past 10-year sales data.
In this article, we discuss what should be forecast, forecasting types, and forecasting methods. Forecasting is obviously a challenging activity. In future articles, I will explore some common forecasting methods including exponential smoothing methods, ARIMA models, dynamic regression models, LSTM, etc.
“Do you see these great buildings? Not one stone will be left upon another which will not be torn down.” (Mark 13:1–2)
