Polling Industry 101: The Missing "Don't Knows"
AKA why the reported figures aren't necessarily the whole picture
The Smokeless Room is a newsletter by Rushaa Louise Hamid designed to help you clear the smoke from the air and better understand the tools of decision makers, with a special focus on all you need to know to sift the bunk from the gold of opinion surveys!
A Quick Note
Before we break down an aspect of polling that causes a lot of confusion, I just wanted to flag that the British Polling Council recently made a guide aimed at journalists to highlight what to look out for when reporting on polls. The contents cover briefly some of the same topics explored here, though the angle is different, and it's certainly worth a browse!
One of the biggest problems with looking at polling figures is how the media choses to - or not to - report the amount of people who don’t have an opinion.
Most political polling figures that you will see are weighted by likelihood to vote, with don’t knows and refusals excluded. By eliminating those who said “Don’t know” or refuse to give an answer it can give a clearer picture, especially when weighted. However it also can give a false perspective of how a race is shaping up, particularly if we can’t see just how many people have yet to declare their decision.
Let’s take the below figures - the top one includes don’t knows whilst the bottom excludes them.
The without figures are how the race would be presented in media. It seems like the Red Party is surging ahead, cruising for an easy win, even though half of all people asked haven’t given an answer yet.
Looking again a month later it seems like things have dramatically changed when we look at the without figures:
Not getting to see the original figures it appears that the Red Party have shed a dramatic amount of their support - in actual fact it’s held steady. Rather as people are deciding who they will pick they seem to all be opting for the Blue Party. The story with and without the don’t know information is very different, transforming from one where Red members are driving away supporters to one where the Blue Party is efficiently scooping up undecided voters.
So why not include these figures always?
There are a number of reasons someone may be listed as “Don’t know”:
They do know who they would vote for but they don’t want to say.
They are 99% certain they’d vote for a candidate but feel bad because they haven’t done research into it yet and hope that they will before election day.
They are split between two candidates.
They are planning to tactically vote but they don’t want to say their tactical vote instead of their ideal vote, but they also don’t want to lie.
They are planning on spoiling their ballot.
They honestly have no idea who they are going to pick.
You’d be surprised at just how many people genuinely don’t know. A large amount of voters I’ve spoken to throughout my life decide when they’re literally in the voting booth.
However as you can see from above, not all the don’t knows are actually unsure - and as a result we can’t always infer that “don’t know” really means “don’t know” all the time. There is also the strong likelihood that if you asked people who they wouldn’t vote for they’d have some pretty concrete ideas. Some of the recent theories about Trump overperforming relative to polls centre on the idea that some voters might have been too embarrassed to declare their vote for him (this phenomenon is also where you get the term “shy Tory” from).
So why exclude them at all? Mostly this is centred in the need to have a neat narrative, especially with the fact that when it comes down to actually voting there’s not really an option to vote with a “don’t know” (spoilt ballots aside). If you want to be the polling company with the closest figures to the results you can’t just have figures including don’t knows in.
When it comes to early in a race though it is often worth digging deeper to see if there is a sizeable portion of people who for whatever reason are not declaring their preference - it can help better explain later changes and doesn’t pretend that answers people give are always neat.
Next week we’ll be continuing to look at other aspects of the polling industry!
As always, if you'd like to drop me a note, you can contact me by replying to this email or over on Twitter at @thesecondrussia.