The problem with polls
Veteran pollster Peter Kellner has a fascinating article in the Politics Counter today.
The article is about why he thinks Remain is more likely to win the EU Referendum and most people reading it will probably focus on that conclusion but actually what he says about why opinion polls are often wrong is in some ways even more interesting.
I had a twitter exchange with a journalist on the North West Evening mail this week, after taking exception to the results of one of the voodoo polls in the paper.
Basically any newspaper website, or any other website, which simply asks readers to click on their view without making any effort to check for things like multiple voting is going to produce totally unreliable results and would be anywhere near correct only by great good fortune in the same way that a broken clock is right twice a day.
That's why Mike Smithson of Political Betting calls such polls "Voodoo polls" and he's right,
The interesting thing about today's Kellner article is that he addresses the difficulty that both online and phone polls are addressing in terms of how hard it is to reach a representative sample of voters, and as he says, the amazing thing is not that the opinion polls are sometimes badly wrong as how often they are right.
Here's an extract from the article
"ALL POLLS HAVE A PROBLEM
Before we discuss these differences, a broad point needs to be made about all polls. They face a huge challenge getting anything right. By definition, online polls survey people who have chosen to join a polling panel.
In theory, telephone polls are able to reach everyone who has a phone. But they are at the mercy of response rates; and in recent years these have collapsed. Twenty years ago, according to one of our most respected telephone pollsters, they achieved a response rate of around 30%. To obtain 2,000 interviews, they asked their computer to generate 7,000 random residential telephone numbers. These days, to obtain 2,000 interviews, they need to start with as many as 28,000 phone numbers, for response rates have fallen to just 7%.
In short, all polls require research companies to extrapolate from the small proportion of the general public they can reach to the far larger number of people who neither join online panels nor respond to telephone polls. The surprising thing about is not that today’s polls sometimes get things wrong, but that they have such a good record of getting so many things right.
The reason why our leading polling companies are so good, so often is that they go to great lengths to make their samples match the country’s population – by age, gender, region, social class, past vote and so on. Usually this process generates accurate results. But sometimes it doesn’t, either because the silent, unpolled, majority, differs from the poll-friendly minority in some way that is not captured even by the smartest demographic sampling – or because of “mode effects”, in which the way a poll is conducted prompts some respondents to conceal their true feelings."
Seriously, for anyone who wants to understand the workings of opinion polls, Peter Kellner's article is an absolute must read. You can read it here.
The article is about why he thinks Remain is more likely to win the EU Referendum and most people reading it will probably focus on that conclusion but actually what he says about why opinion polls are often wrong is in some ways even more interesting.
I had a twitter exchange with a journalist on the North West Evening mail this week, after taking exception to the results of one of the voodoo polls in the paper.
Basically any newspaper website, or any other website, which simply asks readers to click on their view without making any effort to check for things like multiple voting is going to produce totally unreliable results and would be anywhere near correct only by great good fortune in the same way that a broken clock is right twice a day.
That's why Mike Smithson of Political Betting calls such polls "Voodoo polls" and he's right,
The interesting thing about today's Kellner article is that he addresses the difficulty that both online and phone polls are addressing in terms of how hard it is to reach a representative sample of voters, and as he says, the amazing thing is not that the opinion polls are sometimes badly wrong as how often they are right.
Here's an extract from the article
"ALL POLLS HAVE A PROBLEM
Before we discuss these differences, a broad point needs to be made about all polls. They face a huge challenge getting anything right. By definition, online polls survey people who have chosen to join a polling panel.
In theory, telephone polls are able to reach everyone who has a phone. But they are at the mercy of response rates; and in recent years these have collapsed. Twenty years ago, according to one of our most respected telephone pollsters, they achieved a response rate of around 30%. To obtain 2,000 interviews, they asked their computer to generate 7,000 random residential telephone numbers. These days, to obtain 2,000 interviews, they need to start with as many as 28,000 phone numbers, for response rates have fallen to just 7%.
In short, all polls require research companies to extrapolate from the small proportion of the general public they can reach to the far larger number of people who neither join online panels nor respond to telephone polls. The surprising thing about is not that today’s polls sometimes get things wrong, but that they have such a good record of getting so many things right.
The reason why our leading polling companies are so good, so often is that they go to great lengths to make their samples match the country’s population – by age, gender, region, social class, past vote and so on. Usually this process generates accurate results. But sometimes it doesn’t, either because the silent, unpolled, majority, differs from the poll-friendly minority in some way that is not captured even by the smartest demographic sampling – or because of “mode effects”, in which the way a poll is conducted prompts some respondents to conceal their true feelings."
Seriously, for anyone who wants to understand the workings of opinion polls, Peter Kellner's article is an absolute must read. You can read it here.
Comments