This is the second post of a two-part series on understanding projections. The first post, which explains why projections currently disagree on which party is leading, is available here.
The chances that any given projection model will hit the exact seat count is close to zero. So if a projection tells you that party A leads party B by 20 seats, is it a close race? Or is party B basically out of the running?
This post provides a back-of-the-envelope argument for determining the uncertainty of the national seat count of the three main parties for the 2015 election. The main conclusions are:
- For this election, the standard error on the main parties' seat counts is roughly 15-20 seats. This implies that the 95% confidence interval should extend 30-40 seats in either direction, and that the standard error on the difference between two parties is about 30 seats. Thus, a 20-seat lead by a party means a lead of about 2/3 of a standard deviation, which translates to a roughly 75% chance of winning (if the 3rd party is not in contention) assuming a normal distribution. The approximate probabilities that I have been providing in my posts are based on this rough calculation. (Currently, I think that the Liberals have a 15% chance of winning if you take my adjusted projection as your best guess, and 40% chance if you take my unadjusted projection as your best guess.)
- Of the models providing seat ranges and probabilities, The Globe's Election Forecast gives the most reasonable estimates. Too Close to Call, The Signal and Le calcul électoral also run simulations to account for uncertainty, but their simulations appear to be miscalibrated and most likely understate the extent of the uncertainty. (I should mention that Too Close to Call's riding-level uncertainties are credible and highly recommended.) ThreeHundredEight's ranges are not based on simulations, and are difficult to interpret in terms of probabilities. (Links to all of these websites are available on the left.)
Where does the "15-20 seats" come from?
As we know, polls are not exact. Part of the issue is sample size, which determines a poll's reported "margin of error." But a greater issue is turnout: pollsters in Canada do not have a good handle on who is actually doing to show up. (There's also movement in voting intentions as a poll is being conducted.) Therefore, even very large polls and their aggregation come with a significant degree of uncertainty.
How much is an average of polls likely to be wrong? Helpfully, ThreeHundredEight.com's methodology page tells us that in recent Canadian elections, its poll average was off by an average of 2.15 points per party. How does this relate to the standard deviation for main parties in a federal election?
- For a normal distribution, the average deviation is roughly 80% of the standard deviation. So this would bump up the estimate of the standard error to 2.7 points.
- Standard errors for parties closer to 50% are bigger than for smaller parties. So the 2.7 points estimate understates the standard error for large parties.
- There are more polls in a federal election, which reduces the sample-size related error.
I would venture that the latter two effects roughly cancel each other out, so the standard deviation on the support of the Liberals, Conservatives and NDP is probably, say, 2.5 to 3 points.
If the parties' support levels were independent, to get the standard deviation on the difference between two parties' support, one would multiply the above numbers by sqrt(2). But independence is clearly violated: if a party outperforms polls, other parties are likely to underperform them. Therefore, the true standard deviation on the difference between two parties is a bit bigger: roughly 4 to 5 points, if you believe the assumptions made so far.
How many seats is that? At most times during this campaign, there have been 50-55 races decided by under 4 points, and 65-70 races decided by under 5 points (though these figures are currently a bit higher). Roughly speaking, each of the main parties is winning 30% of them, and losing 30% of them. Thus, I estimate that a party stands to gain/lose 15-20 seats on a one-standard-deviation error.
What about other sources of uncertainty?
There are indeed other sources of uncertainty, e.g. arising from the transposition of regional vote shares into seats. But they are relatively small, and roughly independent from the uncertainty on the national vote share. Therefore, accounting for them would add little to the estimate above.
(Clarification: If the standard error from poll inaccuracy/noise is 18 (variance 324), and the standard error from independent factors is, say, 7 (variance 49), then the total standard error would be roughly 19.3 (variance 324+49=373), not much more than the 18 from polls.)
There are so many assumptions in the above calculations! Is there an independent way of getting the "15-20 seats" estimate?
I've been doing seat projections using a similar methodology since the 2004 election (just for fun, sharing with my buddies, before 2011). Here's how much the final projection was off each time:
- In 2004, the Liberals were underestimated and the Tories were overestimated by around 20 seats.
- In 2006, the Liberals were underestimated and the Tories were overestimated by around 10 seats.
- In 2008, the Liberals were overestimated and the Tories were underestimated by 12-15 seats.
- In 2011, I missed the Liberal, NDP and Tory seat counts by 9 to 14, despite adjustments that improved the accuracy of the model. Indeed, as you can see here, most other models fared worse, sometimes by a lot.
I think it is pretty clear that any claim of a standard deviation under 10 seats, which implies 95% confidence intervals extending less than 20 seats in each direction, is implausibly optimistic. Yet, the confidence intervals provided by Too Close to Call, Le calcul électoral, as well as most of the ones given by The Signal, are this small.
Why are those three sites wrong?
Firstly, I'd like to say that the three sites I mention run simulations to get their ranges, which is in principle much better than the back-of-the-envelope calculations I posted here. The problem is that their simulations appear to be miscalibrated.
- For Too Close to Call, I've had a conversation with Bryan Bréguet, and the problem appears to be that errors in each polling region (Atlantic, Québec, Ontario, MB/SK, Alberta, BC) are treated as fully independent. In reality, a party outperforming the polls in a given region is also more likely to do so in other regions - many factors are common across regions (age of supporters and other socio-economic characteristics, enthusiasm, etc.). Thus, while his simulations yield the right uncertainty in each region, they underestimate the national uncertainty: errors in different regions cancel themselves out more than they do in reality.
- Le calcul électoral does a wonderful job accounting for model uncertainty and statistical uncertainty. (It also deserves props for exemplary transparency in the description of its methodology.) However, it misses turnout uncertainty, which, as explained above, is the dominant problem here.
- Unfortunately, The Signal's methodology is not described in sufficient detail for me to figure out where the problem lies. But its ranges for the national popular vote seem too narrow.