This Blog is based on my talk at the International Symposium on Forecasting ISF2020, titled – Best Practices for Scaling Sales Forecasting.

What Are We Forecasting and Why?
At Wix.com we’ve been using time-series forecasting models as part of our Data Science projects for forecasting Wix’s future collections. This allowed the company two important things: (1) Better budget planning based on future collections; (2) Accurate guidance to the stock market.
Forecasting collections is a challenging task (every forecasting task is) but it’s one we’ve constantly improved on and achieved amazing results. In this blog I want to share some of our insights and practices for scaling a forecasting project.
Scaling Starts from the Bottom – The Importance of Design

For our system to scale and accommodate more features, more models and eventually more forecasters we divided it to the above building blocks. Each building block stands on its own but also knows how to communicate and work with others. A short explanation of each (hopefully I would describe them in more depth at the future, comment below/DM with requests):
- Data Sources -Used to access and query all of our different tables and DB’s.
- Feature Store – Uses the data from the Data Sources to create time-series features. Given a list of requested features it will return a time-series dataset.
- Models – This is included for clarity, references open source models for time-series.
- Models Library – Wrappers for Models, with our choice of hyper-params and compatible to datasets originating from the Feature Store.
- Forecaster – Takes a dataset from the Feature Store and a model from the Model Library and creates a time-series forecaster for our required task.
- Airflow DAG%20as%20code.) – The DAG plays several roles in our system, in this case (more will follow) we’re referring to it scheduling the Forecaster’s run (daily in our case)
As mentioned, this design supports scaling and as important it supports fast experimentation which is the basis of every successful data science project.
Keep Your Friends Close and Your Data Closer
The data is always the most important part of every data science project. When Forecasting a time-series originating from your business, naturally you would want to add ‘inside’ information available to you. When you want to add features to time-series forecasting you have to make sure you know what the future values of those features are (or you’ll have to forecast them as well which adds a lot of complexity). That being said, those features might change over time and might have mistakes of different kinds that will drastically affect your results.
From Downstream bugs to Upstream solutions – keeping a very close look on your results and data is the key to find the downstream problems (more on that in the next section), the focus here is translating these to solutions upstream – based on our design we were able to include data validations on the different layers:
- The Data Sources validates we’re getting the up to date data that fits our expectations.
- The Feature Store validates the features follow our requirements for the dataset.
- The Airflow DAG validates prior to running the Forecasters that the tables were updated and previous processes completed successfully.
Monitoring – How We Adapted to Business Changes
We can divide business changes to two kinds:
- Internal Changes – For example, new sales, new products, pricing changes, etc..
- External Changes – For example a new competitor or even something as far fetched as a worldwide pandemic.
Although the changes might be very different, adapting to them consists of similar stages:
- Spotting the Change – this is where our monitoring comes into place, we’re constantly following our results and compare them to past results through analytics reports and email alerts (again using the DAG).
- Understanding the change – Is there a trend to the change? Can we quantify the impact? Where is it coming from? This is very different between internal and external changes and it’s always a very challenging task. Sometimes even understanding that we don’t quite understand the change could be our conclusion.
- Helping the model grasp the change – This strongly depends on our conclusions from the previous stage, if we have a good understanding of the change and its future behaviour we might introduce it to the model through a feature for example. We might also decide to see if the model will pick up on the change by itself.
When and How to Improve Your Models

If you followed carefully, by now you have a pretty good system, but now what? When should you improve your model and how? Brace yourself, this section would include tough questions and not as many decisive answers.
That being said, one answer can rule them all – experimentation is the key to handling each of these issues.
If it ain’t broken, don’t fix it?
The Forecaster is working good, but you start thinking of a new feature that should be very relevant – should you add it?
If it starts to squeak, then fix it?
The Forecaster’s error is slightly increasing, you’re going back to the previous section and marking a few possible changes. But you’re not sure how this changes in Forecaster would turn out – Should you add them?
When is change a good change?
Changing the Forecaster or adding a feature adds complexity and possible future technical debt (although our system is designed to reduce that). What is the impact of this new feature? Would this impact increase over time? Is the improvement significant enough to justify the cost of adding a new feature or changing something in the forecaster?
Following up on the change and rewriting history
Let’s say we’ve added the new change, now is the time to go back to monitoring the change and seeing its effect. But in a sense, we’re now comparing two different Forecasters (Apples vs. Oranges?). On the one hand, we want to see our new Forecaster vs. the old, but on the other hand, we want to see our new Forecaster compared to the new Forecaster as if it ran, for example, a week ago (hereby apples vs. apples).
For that we have two tables:
- Real history which is basically the live results of the model in production for that day.
- Updated History the history as if our current model with the current features and data was with us all along (we’re running all the previous dates for that).
Conclusions
Scaling and keeping your forecasting system accurate while the business is rapidly changing is a very challenging task but using these practices has gone a long way for us and hopefully, it will serve you right as well.