Updated to reflect change of lead in Montmagny--L'Islet--Kamouraska--Rivière-du-Loup
This post will compare the accuracy of my final projection with that of others that were freely available during the campaign. For a region-by-region analysis of how my projection fared, click here. Once again, the websites mentioned here are linked through the left bar.
Here are the eight final projections that were based on an average of polls (C-N-L-B-G-I):
148-100-44-14-1-1 (The Mace)
152- 94-46-15-0-1 (Canadian Election Watch)
142-114-39-12-0-1 (Riding by Riding)
151- 91-47-18-0-1 (Calgary Grit)
144- 98-51-15-0-0 (LISPOP)
155- 86-47-20-0-0 (democraticSPACE)
143- 93-58-14-0-0 (Too Close to Call)
143- 78-60-27-0-0 (ThreeHundredEight.com)
First, I would like to congratulate democraticSPACE for being the only model projecting a Conservative majority. On that front, I fared honourably by having the second highest Conservative seat count. Four websites estimated the probability of a Conservative majority. Obviously, democraticSPACE had the highest, right around 50%. Canadian Election Watch said it would occur with a 45% chance, while Calgary Grit had 28%, and The Mace, 13%.
Although the Conservative seat count was the most politically relevant figure to project, to get a sense of the overall quality of a projection, one needs to look at the other numbers as well. Below is half of the total deviation of each of the above eight projections. (This is the sum of the absolute value of the difference between the projected and the actual result for each of the parties and independents. It is divided by 2 because, by definition, it is always even.)
21 (The Mace)
24 (Canadian Election Watch)
25 (Riding by Riding)
28 (Calgary Grit)
28 (LISPOP)
29 (democraticSPACE)
34 (Too Close to Call)
49 (ThreeHundredEight.com)
Once again, Canadian Election Watch comes in a strong second, this time behind The Mace. democraticSPACE actually did not do too well, placing sixth. Interestingly, the two prognosticators with newspaper columns fared worst... (Of course, Éric and Bryan still have great websites with interesting content.)
If you average the above two measures of success (Conservative seat count and half total deviation), Canadian Election Watch comes in first! I'm not going to use this to declare victory, but I think there's a strong case for saying that I gave at least as accurate a portrayal of the overall situation as any other projection.
I also note that EKOS, which projected 138-113-41-15-1, has a half total deviation of 28, in the same ballpark as the projections above based on multiple polls. Update: Steven Britton also projected on the EKOS poll alone, and got 150-116-37-5, which has a half total deviation of 17, better than the above projections. However, his official call, below, was less accurate.
Now, it is true that seat projections based on polls did not do too well due to poll inaccuracy. Would relying on intuition and other information have been better? To look at that, I compiled the following relatively better publicized projections that were obtained via other methods:
151-86-45-24-1-1 (Andrew Coyne)
156-76-46-30-0-0 (Steven Britton)
146-83-55-22-1-1 (Dan Arnold, aka Calgary Grit)
152-71-52-31-0-2 (Bernard von Schulmann, aka BC Iconoclast, April 25)
146-65-63-33-0-1 (Election Prediction Project)
156-46-60-46-0-1 (Glen McGregor, April 29)
Two of these called for a bare Conservative majority, but in both cases, the NDP was very low, and the Bloc was absurdly high. The half deviations were as follows:
32 (Andrew Coyne)
38 (Steven Britton)
40 (Dan Arnold, aka Calgary Grit)
47 (Bernard von Schulmann, aka BC Iconoclast, April 25)
59 (Election Prediction Project)
69 (Glen McGregor)
The striking thing here is that everyone did worse than most projections based on polling averages. Thus, while relying on polls is far from perfect, it still gives us a better idea of what's going on than letting "gut" and "instinct" cloud one's judgment. Obviously, some prognosticators probably did predict 166 Tory seats (one of the commenters here was close, with 170). However, the above numbers suggest that in most cases, taking poll numbers seriously, even when they're significantly off, is still helpful.
What about riding-by-riding predictions? As far as I know, 7 of the above projectors bothered making a call for each of the 308 races (democraticSPACE also did so for most of them, but did not call some close races). Here is the number correct for each:
259 (Canadian Election Watch)
255 (Riding by Riding)
250 (-1?) (Steven Britton)
242 (Too Close to Call)
234 (ThreeHundredEight.com)
234 (Election Prediction Project)
217 (Glen McGregor)
Once again, Canadian Election Watch performs strongly, this time coming out on top. I'm particularly proud of my projections for BC, where I registered 34/36. The two predictions that relied the least on polls were the worst. In fact, the four most widely known ones came out at the bottom. So while everyone, myself included, did pretty poorly, we still did better than the media will give us credit for.
So where does this leave us? As I've emphasized above, while these results are far from satisfying for election projectors, they still point to the value of relying on the hard data provided by imperfect polls. None of the 7 "soft" (i.e. not based on a numerical model) projections had the Bloc below 22 seats, while 5 of the 8 projections using models had it at 15 or fewer.
As for myself, I'm obviously not thrilled about the absolute result, but very satisfied about how I fared compared to others. Moreover, I'm glad that some of the major issues emphasized on this blog and little discussed elsewhere, such as massive Conservative gains the GTA area, the efficiency of the NDP surge in Québec and the ballot box penalty for the Bloc, all came to pass.
It was my goal to provide you with the most accurate and up-to-date coverage of voter intentions and seat implications for the 41st General Election, and I believe that in many ways, I have succeeded. I hope that you agree, and that you will check back sporadically over the next few years for more political coverage and insights. In fact, please check back over the next few days: I will be analyzing the possible implications of these results as I digest them.
8 comments:
Well, then, I've got to compliment myself on my 160-104-33-11 prediction. I think it's somewhere in your comments and I did tweet it on Sunday sometime. Well done, me!
Yeah, with a half total deviation of 9, that's by far the best of anything out there. Well done! I hope that some of the things mentioned here helped :)
I'm curious where you plan to go from here, and if you got my e-mail asking the same?
Sorry, didn't get that email. I will continue to provide seat projections whenever a poll comes out, though I suspect that we'll only get one or two a month for the next while. As before, I will continue having occasional posts on economic policy. I will also closely cover the changes that are coming to the House of Commons: the Tories will surely seek to give Québec fewer seats than warranted by its population, and that will pass.
In terms of elections, I may provide some coverage of Quebec and BC provincial votes, and of US Presidential and Congressional elections. If Harper isn't careful, I may also have to cover a Quebec separation referendum in 2014...
How about you?
There is the Ontario provincial election in October, I know another projection site plans to cover that as well.
I dont think anyone would have been able to predict a lot of what just happened, no pollster could face the public and tell them that the bloc would win 4 seats or 2 leaders will lose their seats.
Good analysis. My error in my prediction was based in my assumption the voters in Quebec were more directly engaged and were learning about their candidates. I was expecting a worse result simply because as things like Ruth Ellen Brossaeau's trip to Las Vegas, the former Communist, and so on, I expected the NDP vote to drop a bit, with the voters returning to the Bloc.
Sadly, I overestimate people, sometimes.
I also published a purely mathematical prediction of the results based on bumping CPC support up 3% and the other three national parties down by 1% each.
The results I got were CPC 159, NDP 114, LPC 29, Bloc 6.
The lessons: 1) Do not overestimate the voters and their engagement in the election.
2) Do not use intuition, because it's usually wrong.
3) Polls are mostly rubbish. Especially Ekos where the election results were way outside their margin of error.
4) A prediction is only good as the data fed into it.
I am going to spend some time looking at the ridings I got wrong and see if I can find a pattern and a reason, or, at least attempt to build a "confidence factor" into the prediction to produce a range of values for the final totals in my prediction.
I'm not sure Quebecers were so uninformed about the candidates in the first place. Many people voted for NDP candidates that they knew were not up to par because they wanted to vote for the NDP.
It's like how Lethbridge elected a Conservative that hid from the media the entire time, and literally ran into the bathroom when a journalist found him.
So I'd change your lesson #1 to: Do not overestimate the importance of the local candidate and underestimate the importance of the party. Also, I wouldn't condemn polls so strongly: even EKOS would have been quite close had they used their voter commitment index for screening. But I agree with your other points.
Good luck building in a "confidence factor." The only two websites that have done so are CalgaryGrit and The Mace, and both of their confidence ranges are unrealistically narrow. Looking at the past 4 elections, the standard deviation of a poll-based projection for the #1 or #2 party should be around 12-15, so the width of a 95% confidence interval should be 50-60 seats. (Yes, in this sense, polls are rubbish!)
The confidence factor I'm talking about isn't so much based on the confidence interval of the poll upon which my prediction is based, but more along the lines of a combination between (a) the difference between first and second place in the last election, (b) the projected difference between first and seond place, (c) popularity of the incumbent, and (d) popularity of the challenger(s).
No projection method is perfect; but I think there's a way to improve it. Ideally, I'd like to see the election result fall within the predicted range for all parties.
Post a Comment