This is Why You Should Ignore IHME’s Forecasts
IHME is miserably misleading us about where the pandemic is heading. See if you agree the data proves it.
On Friday, the Institute for Health Metrics and Evaluation (IHME) released three new fatality forecasts. Their most likely scenario stated that 410,000 people will die from Covid-19 in the US by January 1. These forecasts have already been covered extensively by major news media (CNBC, NBC, NPR, USA Today, San Jose Mercury News, etc.).
There’s only one problem. IHME’s forecasts in the past have been inaccurate by as much as several hundred percent. In many cases, their forecasts have literally been worse than no forecasts at all because they created such misleading ideas about where the pandemic was headed.
IHME’s forecasts for January are no better. Indeed, they are so completely unrealistic that I wonder whether they are really based on science, or whether the science has become secondary to some other agenda.
In this article, I will provide data on the inaccuracy of IHME’s earlier forecasts, and I will give you additional reasons to ignore their forecast for January.
The Data: IHME’s Record on State Forecasts
I lead one of about 20 groups that submits fatality forecasts to the CDC for use in the CDC’s Ensemble model, which is the forecast of record for the CDC. IHME also contributes, along with groups from Carnegie Mellon, Columbia, Johns Hopkins, MIT, University of Massachusetts Amherst, UCLA, and several others.
This is elite company, and in this elite company, IHME’s forecasts look mediocre at best.
I recently completed an evaluation report of the state-level fatality forecasts that were submitted to the CDC on August 3 for the period from August 2–29. For this period, IHME received an average grade of C-; it was significantly below the leading groups for overall forecast accuracy.
Figure 1 shows IHME’s performance state by state for the cumulative Week 1–4 state-level forecasts submitted to the CDC on August 3, i.e., for the period from August 2–29.
An accurate forecast model would have nearly all of its points between the two dark ±25% lines in the middle. IHME misses that target more than half the time. One quarter of IHME’s forecasts missed by more than 100%.
As mediocre as these results were, they were actually an improvement over IHME’s prior performance. For the comparable period in July (July 5-August 1), IHME averaged 191% error in its four-week state-level death forecasts. As Figure 2 shows, IHME was in the bottom half of the CDC forecast groups for the period and was markedly worse than the best groups.
As IHME’s scatter plot for the July period shows (Figure 3), more than one-third of its forecasts missed by at least a factor of 2. Neither its worst high forecasts nor its worst low forecasts are shown on the graph because they are outside the ±500% error range shown on the graph.
IHME’s July forecasts were so bad that — get this — if you take their August forecasts and apply them to July, those forecasts actually do better than their July forecasts did. Figure 4 shows what that looks like.
When your forecasts for the wrong period are more accurate than your forecasts for the right period, it’s time to admit that you aren’t very good at forecasting.
More Data: IHME’s Record on National Forecasts
At the national level, IHME’s forecasts compare to the other forecast models as shown in Figure 5, which shows the national forecasts for July 5-August 1. The goal is to be close to the black 0% line in the middle. It’s true that other groups did worse, but that doesn’t mean IHME’s forecast was accurate. IHME’s forecast missed by almost 50%, and the upper end of their forecast range was more than 25% low.
For the week just ended (week ending September 6), IHME’s most recent forecasts are performing worse than most groups’ forecasts (Figure 6). IHME’s forecast missed by 20% in just one week; most other forecast groups were much more accurate.
None of this is new. IHME’s forecasts have been poor from the beginning of the pandemic. A review of IHME’s 95% prediction interval forecasts in April found that anywhere from 49 to 73% of actual death counts lay outside their intervals, depending on the forecast. That number is supposed to be 5%. As the review commented, this is an order of magnitude worse than it should be.
IHME’s forecasts for one week missed by 20%. Their forecasts for one month missed by 50%. How much more will their forecasts miss by January, which is still four months away? It’s easy to imagine them being inaccurate by a hundred percent or more.
What’s Behind IHME’s January Forecast?
One reason IHME’s forecasts are so poor is that their assumptions border on the absurd. Let’s look at what would need to happen for IHME’s 410,000 death forecast to come true.
First, their forecast requires the US to pack more deaths into the next four months than we’ve had since the beginning of the pandemic. We would have to average more than 1800 deaths per day for each of the next 115 days — which is 100 more days than we’ve had with that many deaths so far.
Second, the long-term trend from the past six weeks would need to reverse immediately. The daily death rate would need to increase to 225% of what it was last week, despite the fact that the death rate has been trending down about 10% per week in recent weeks, and the fact that positive tests have also been trending down sharply since mid-July (Figure 7).
Third, for IHME’s most likely scenario to occur, for the next four months public officials will need to avoid noticing that deaths have more than doubled, and they will need to take no corrective action. Public officials will need to persistently ignore reports from scientists, ignore alarming stories in the media, ignore criticism from the other political party in an election year, and ignore increasingly loud complaints from their constituents that the death rate is doubling.
Does that sound plausible to you? Does it sound most likely? Of course not.
This is not going to happen.
What could possibly cause IHME to think it will?
Is Science IHME’s Only Consideration?
There are competent scientists at IHME. Can competent scientists really convince themselves that the whole country will go to sleep for four months and do nothing while the death rate doubles — in their most likely scenario?
That scenario isn’t most likely. It isn’t even just unlikely. It’s virtually impossible.
So why would the scientists at IHME present that scenario as most likely?
I believe there are two issues.
First, as the data in this article has shown, they really are bad at forecasting. They need to improve.
Second, I don’t think they really believe their (bad) forecasts either. They can’t really believe that deaths will increase to 410,000 by January 1, while government officials do nothing. Rather, I believe they are using their forecast as a cattle prod to push government officials to ensure that deaths don’t increase to that level. If that’s true, they have ceased to function as scientists and instead have begun to function as a political action group.
This is the last thing the world needs right now. We already have a substantial portion of the public questioning the science or just outright disbelieving it. In truth, these people have a point. Much of the science of coronavirus isn’t “settled.” Scientists have many different theories and opinions. There are still many “unknowns” or “partially knowns.” In this context, it is not helpful to have a group of scientists blurring the issue even further by presenting their political action agenda under the guise of science.
IHME needs to develop a level of forecasting skill that’s commensurate with the level of attention their forecasts have received. And while they’re at it, they should remember that, if we want people to have any faith in science, scientists need to stay scientific and leave the politics to the politicians.
More Details on the Covid-19 Information Website
I lead the team that contributes the CovidComplete forecasts into the CDC’s Ensemble model. For more forecasts and US and state-level data, check out my Covid-19 Information website.
My Background
For the past 20 years, I have focused on understanding the data analytics of software development, including quality, productivity, and estimation. The techniques I’ve learned from working with noisy data, bad data, uncertainty, and forecasting all apply to COVID-19.
