Projecting Confidence

How the Probabilistic Horse Race Demobilizes the Public

Inspired by Donald Trump’s shocking victory over Hillary Clinton in the 2016 general election, Sean Westwood, Yphtach Lelkes and I set out to interrogate the question of whether elecion forecasts — particularly probablistic forecasts — might have helped to create a false sense of confidence in a Clinton victory, and ultimately led many on the left to stay home on election day.

Clinton herself was quoted in New York Magazine after the election:

I had people literally seeking absolution… ‘I’m so sorry I didn’t vote. I didn’t think you needed me.’ I don’t know how we’ll ever calculate how many people thought it was in the bag, because the percentages kept being thrown at people — ‘Oh, she has an 88 percent chance to win!’

Is it plausible that forecasting could have affected the election?

For this to affect the election, it has to be out there in the media reaching potential voters, depress turnout, and to affect Clinton supporters (and/or Clinton campaigners) moreso than for Trump.

In 2016 there was significantly more media covrerage of probablistic forecasts than in past election cycles:

(Number of articles mentioning probabilistic forecasts indexed by Google News.)

And our research (see results below) suggests that candidate who is ahead in the polls is more affected by probablistic forecasts. In 2016, that was Hillary.

But irrespective of 2016, when you look at who engages with this material in both social media and television media, it’s outlets with a left-leaning audience. The websites that present their poll aggregation results in terms of probabilities have left-leaning (negative) social media audiences — only realclearpolitics.com, which doesn’t emphasize win-probabilities, has a conservative audience:

These data come from the average self-reported ideology of people who share links to various sites hosting poll-aggregators on Facebook, data that come from this paper’s replication materials.

When you look at the balance of coverage of probabilistic forecasts on major television broadcasts, there is more coverage on MSNBC, which has a more liberal audience.

How much influence do forecasters really have?

Well, FiveThirtyEight’s 2018 coverage seems to have been highly influential. After their real-time forecast had GOP’s odds of taking the House spiking at 60% at around 8:15PM, PredictIt’s odds on the GOP rose above 50–50, & U.S. government bond yields saw brief spike of 2–4 basis points.

This spike seems to have occurred because a number of big, Republican-dominated districts started reporting returns before those that went toward Democrats and because it was making inferences from partial vote counts:

This was first reported by Colby Smith & Brian Greeley of FT.com. They report that because markets expected to see more inflation under a Republican House (high spending, low taxes) the U.S. Bond yield rose.

Was this just a correlation? Possibly, but there was pretty much nothing else happening in the U.S., and it was like 1 am in Europe, as pointed out in the FT.com piece above.

Josh Tucker suggested that 538 might be driving prediction markets back in 2012 in a Monkey Cage blogpost.

Our research on forecasting and perception

Our research shows that probablistic election forecasts make a race look less competitive. Participants in a national probability survey-experiment were substantially more certain that one candidate would win a hypothetical race after seeing a probablistic forecast than after seeing the equivalent vote share estimate and margin of error. This is a big effect — those are confidence intervals not standard errors, with p-values below 10^-11.

Why do people do this?

More research is needed here but we do have some leads. First, small differences in the election metric most familiar to the public — vote share estimates — generally correspond to very large differences in the probability of a candidate’s chance of victory.

Andy Gelman referenced this in passing in a 2012 blogpost questioning the decimal precision (0.1 percent) that 538 used to communicate its forecast on its website:

That’s right: a change in 0.1 of win probability corresponds to a 0.004 percentage point share of the two-party vote. I can’t see that it can possibly make sense to imagine an election forecast with that level of precision…

Second, people sometimes confuse probabilistic forecasts with vote share projections, and incorrectly conclude that a candidate is projected to say win 85% percent of the vote, rather than to having an 85% chance of winning the election. About 1 in 10 peope did this in our experiment.

As Joshua Benton pointed out in a tweet, TalkingPointsMemo.com made this very mistake:

Finally, people tend to think in qualitative terms about the probability of events (Sunstein, 2002), (Keren, 1991). An 85% likelihood that something will happen means it’s going to happen. These studies may help explain why after the 2016 election, so many criticized forecasters for “getting it wrong” (see this and this).

What about voting?

Perhaps most critically, we show that probabilistic forecasts showing more of a blowout can lower voting. In Study 1, we find limited evidence of this based on self reports. In Study 2, we show that when participants are faced with incentives designed to simulate real world voting, they are less likely to vote when probabilistic forecasts show higher odds of one candidate winning. Yet they are not responsive to changes in vote share.

Could this actually affect real world voting?

Consider 2016 — an unusually high number of Democrats thought the leading candidate would win by quite a bit:

And people who say the leading candidate will win by quite a bit in pre-election polling are about three percentage points less likely to say they voted after the election than people who say it’s a close race. That’s after controlling for election year, prior turnout, and party identification.

The data here are from the American National Election Study (ANES) and go back to 1952.

Past social science research also provides evidence that the perception of a close race boosts turnout. Some of the best evidence comes from work that analyzes the effects of releasing exit polling results before voting ends, which clearly removes uncertainty. Work examining the effects of East Coast television networks’ “early calls” for one candidate or another on West Coast turnout generally find small but substantively meaningful effects, despite the fact that these calls occur late on election day (Delli Carpini, 1984), (Sudman, 1986). Similar work exploiting voting reform as a natural experiment shows a full 12 percentage point decrease in turnout in the French overseas territories that voted after exit polls were released (Morton, Muller, Page, & Torgler, 2015). These designs are not confounded with the tendency for campaigns to invest more in campaigns in competitive races.

Researchers consistently find robust correlations between tighter elections and higher turnout [see (Geys, 2006); (Cancela & Geys, 2016) for reviews]. Furthermore, (Nicholson & Miller, 1997) provide evidence from statistical models that prior election returns also explain turnout above and beyond campaign spending, particularly when good polling data is unavailable.

Field experiments provide additional evidence that perceptions of higher electoral competition increases turnout. This work finds substantive effects on turnout when polling results showing a closer race are delivered via telephone [among those who were reached, (Biggers, Hendry, Gerber, & Huber, 2017)] but null results when relying on postcards to deliver closeness messages [for which it’s not possible to verify the treatment was actually read, (missing reference); (Biggers, Hendry, Gerber, & Huber, 2017). Finally, one study conducted in the weeks leading up to the 2012 presidential election found higher rates of self-reported, post-election turnout when delivering ostensible polling results showing Obama neck-and-neck with Romney [which was not consistent with the extant polling data showing a comfortable Obama lead, (Vannette & Westwood, n.d.)].

Could this affect politicians as well?

Candidates’ perceptions of the closeness of an election can affect campaigning and representation (Enos & Hersh, 2015), (Mutz, 1997).

These perceptions can also shape policy decisions — -for example, prior to the 2016 election, the Obama administration’s confidence in a Clinton victory was reportedly a factor in the muted response to Russian intervention in the election.

And former FBI Director James Comey, because of his confidence in a Clinton victory, said he felt that it was his duty to write a letter to Congress on October 28 saying he was reopening the investigation into her emails. Comey explained his actions based on his certain belief in a Clinton win: ‘‘[S]he’s gonna be elected president, and if I hide this from the American people, she’ll be illegitimate the moment she’s elected, the moment this comes out’’ (Keneally, 2018). Nate Silver at one point said ‘‘the Comey letter probably cost Clinton the Election.’’

Media coverage Washington Post, FiveThirthyEight’s Politics Podcast, New York Magazine, Political Wire.

References

1. Keneally, M. (2018). Comey says everyone — himself included — thought Clinton would win 2016 election. ABC News. ABC News Network. Retrieved from https://abcnews.go.com/Politics/comey-included-thought-clinton-win-2016-election/story?id=54486869
2. Biggers, D. R., Hendry, D. J., Gerber, A. S., & Huber, G. A. (2017). Experimental Evidence about Whether (and Why) Electoral Closeness Affects Turnout. Retrieved from https://huber.research.yale.edu/materials/67_paper.pdf
3. Cancela, J., & Geys, B. (2016). Explaining voter turnout: A meta-analysis of national and subnational elections. Electoral Studies, 42, 264–275.
4. Enos, R. D., & Hersh, E. D. (2015). Campaign Perceptions of Electoral Closeness: Uncertainty, Fear and Over-Confidence. British Journal of Political Science, 1–19.
5. Morton, R. B., Muller, D., Page, L., & Torgler, B. (2015). Exit polls, turnout, and bandwagon voting: Evidence from a natural experiment . European Economic Review , 77, 65–81. https://doi.org/http://dx.doi.org/10.1016/j.euroecorev.2015.03.012
6. Geys, B. (2006). Explaining voter turnout: A review of aggregate-level research. Electoral Studies, 25(4), 637–663.
7. Sunstein, C. R. (2002). Probability neglect: Emotions, worst cases, and law. The Yale Law Journal, 112(1), 61–107.
8. Nicholson, S. P., & Miller, R. A. (1997). Prior beliefs and voter turnout in the 1986 and 1988 congressional elections. Political Research Quarterly, 50(1), 199–213.
9. Mutz, D. C. (1997). Mechanisms of Momentum: Does Thinking Make It So? The Journal of Politics, 59(1), 104–125.
10. Keren, G. (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77(3), 217–273.
11. Sudman, S. (1986). Do Exit Polls Influence Voting Behavior? Public Opinion Quarterly, 50(3), 331–339.
12. Delli Carpini, M. X. (1984). Scooping the voters? The consequences of the networks’ early call of the 1980 presidential race. The Journal of Politics, 46(3), 866–885.
13. Vannette, D. L., & Westwood, S. J. Can Pre-election Polls Influence Turnout? Evidence from a Randomized Experiment. Retrieved from https://www.dartmouth.edu/ seanjwestwood/papers/polling.pdf

Originally published at https://solomonmg.github.io/projects/1_project/.