I’ve had a difficult relationship with polls for most of my journalistic career. This reached its apex (or nadir if you look at it that way) years ago when I wrote a column encouraging people to lie to pollsters.
Lying, I argued, would skew the polls and force politicians to make decisions based upon their instincts and their actual principles. I was deluged with angry emails from pollsters at various companies who denounced my attempts to convince people to lie as being “anti-democratic.”
Well, I’m not sure it was anti-democratic, but over the years I have come around to understanding the reason why politicians and others use polls. We live in a big, complex society. If you’re running for national office (or even for a local office that covers a large area) it’s almost impossible for you to speak to every one of your constituents about their concerns.
In this case, polling can provide you with a guide to how the public views particular issues or how they perceive a politician’s efforts to solve important problems or champion important initiatives.
There’s one problem.
People don’t understand polls. People don’t know how they work, how they are taken, who’s doing the polling, how you determine if you can trust a poll or not and how much a particular poll is worth versus another poll.
Over the years I’ve had the opportunity to speak or listen to some of the best minds on polling in Canada and United States. Based on the things that they told me and what I have heard them say, I offer this guide to helping people better understand polls and the roles they play in our democracies.
1) One poll is a snapshot, not a movie
The idea for writing this guide came to me when a friend messaged me on Facebook about a new CNN poll that showed Biden in trouble in battleground states. A devoted Democrat, he was worried that this signaled trouble ahead for the Biden campaign.
Maybe. Maybe not. One poll is a snapshot. Never totally believe what one poll alone tells you. There are so many different conditions that can affect the results of one poll that it’s only a snapshot of a particular moment in a political campaign. This is true whether in Canada or the United States.
What you want to do is look at polling averages – the aggregate of several polls over a period. For instance, I pointed out to my friend that if you look at the average of a number of polls taken in battleground states over the past few months, Biden actually had a lead of 3 to 5 points in many of them. If future polls continue to show trouble for Biden in battleground states, then he should worry.
But one single poll basically means little in the long run.
2) National polls in the United States are useless
Well, maybe not useless, but they don’t mean much. The brains behind Democratic and Republican campaigns pay scant attention to national polls. They are much more interested in state polls. Since America basically elects a president based on the electoral college, a politician can be doing very well in national polls and still lose the election.
Take 2016. Final polls showed Hillary Clinton winning the national popular vote by 2% to 3%, which she did. But since the president of the United States is not elected on a popular vote but by the electoral college, she lost because she did not carry enough states.
It’s a bit different in Canada and Britain where a national poll can give you a better idea of how political parties are doing with public. In Canada you still need to pay attention to the vote in the Prairies versus the vote in Ontario, for instance, but you can trust a national poll more than you can a similar poll in the United States.
Again, that’s only one poll. If a lot of polls show that advantage, then you can take it to the bank.
There are also some important questions that you need to ask about each poll that you consider:
A) Who did the poll?
There are good polling organizations and there are bad polling organizations. It is unfortunate that too many media outlets when referring to recent polls don’t bother to tell you much about the organization that did a poll.
For instance, Ron Elway, the chief political correspondent at NPR, taught me that Quinnipiac was one of the best pollsters in the business especially in its coverage of states like Florida, Ohio and Pennsylvania. Why? They’ve been doing it for a long time and so over the years have ironed out errors in their methods.
You can also put your trust in a Marist poll or Survey USA poll, organizations that have received consistently high marks for their lack of bias in polling and the way they conduct each poll. On the other hand, any poll that you see from TCJ Research or Strategic Vision treat with a huge grain of salt. These polls almost always skew as many as 2-4 points towards Republicans.
B) How many people did they poll?
Randomness is the key to good polling. Normally, to achieve a statistically good sample of the public you need to interview at least 1000 to 1200 participants. That gets harder and harder these days because fewer and fewer people are willing to take part in polls.
When pollsters interview fewer people, the results are harder to trust. That’s why it’s important to pay attention to the statistical error of the poll. For instance if a pollster only interviews 400 to 500 people about a candidate or an important question and the result has a high error rate of 5% or more that means if the poll was something like 49% in favor and 51% against the question, the real result could be anywhere from 54% in favor and 46% against to 44% in favor and 56% against. Those are dramatic differences.
(Note: A friend read this and added something I had forgotten about the margin of error. To quote Pew “A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.”)
A poll with a relatively small number of participants can give you a hint about how people feel about a politician or a particular issue, you just shouldn’t bet your house on the result.
C) Who did they ask?
In a political poll, you want to know if the poll was done of registered voters or likely voters. You get a much truer picture of a campaign when you interview likely voters. These are people who are, as the phrase says, likely to vote in election. Since the United States has such a lousy turn out in elections (be they federal, state, or municipal) somebody can be registered to vote but that doesn’t mean they’re going to go to the polls.
Good pollsters have ways to determine the difference between a likely voter and a registered voter by asking them questions like: did they vote in the last election, have they ever written a letter or an email to their local media, have they ever participated in a campaign, do they know the location of their polling station, etc.
So a poll that says 54% of registered voters support Donald Trump versus a poll that says 54% of likely voters support Joe Biden would be good for Trump but great for Biden.
D) How was the question asked?
The way pollster asks a question will often determine the way the participant will answer it. Good pollsters ask the same question in several different ways to get at the participant’s real views.
Bad pollsters do what are called “push polls” where they ask questions in such a way that they’re pushing you to answer in a fashion that suits the politician or the organization behind the poll.
Politicians and big corporations are infamous for doing push polls in order to produce a result that they like and then spread it among the media, who too often don’t bother to ask how the poll was conducted or how the questions were asked.
E) How was the poll weighed?
Pollsters weigh polls to try to produce a truer result. For instance women and seniors tend to answer the phone more often, which can skew the sex and age ratio of the poll. So they weigh (or adjust) the poll to better match the actual sex and age demographics of the public. Or they might oversample a group to get a more representative result. Good pollsters will include this information in their methodology explanations included at the bottom of each poll.
F) What is the pollsters’ history of success or failure?
I refer you to Nate Silver’s Pollster Ratings where he and his team at fivethirtyeight.com regularly look at the history of each polling organization in terms of successful polling of particular issues, how many polls the pollster has conducted and how the polls are conducted (live, land-line phone, internet, cellphone).
This is a great help to determine the difference between a well-known pollster who has conducted many polls and a fly-by-night organization that pops up to produce polls that favor a particular candidate or issue. It helps to know the success rate of polling predictions because that gives you a better picture of how you should view the poll.
An organization that has a 90% success rate (like Survey USA) should be given greater credence then a poll from an organization that only has a 70% success rate (like Survey Monkey).
G) The shame factor
People do indeed lie to pollsters. This is particularly true in a poll that concerns an unpopular politician or an issue such as racism or misogyny. Or in an exit poll taken the day of an election. If the person being surveyed actually holds racist or misogynist views, they may lie to the pollster about their positions creating a false result.
Pres. Trump benefited from this factor in 2016 particularly after his “grab them by the pussy” remarks. At that time voting for Trump was not seen as a particularly popular thing to do, even though a lot of people wanted to do it. It was also a factor in exit polls. So when a pollster asked if they supported Trump, many people who did said no.
So what does all this tell us about polling? I would recommend you read Nate Silver’s book “The Signal and The Noise” which looks at how polling is done and how predictions made by pundits and politicians often bear no semblance to reality.
Polling may not be a great way to determine how we feel about politicians or controversial issues in, but it may be the best way that we have going forward.