The Smokeless Room is a newsletter by Rushaa Louise Hamid designed to help you clear the smoke from the air and better understand the tools of decision makers, with a special focus on all you need to know to sift the bunk from the gold of opinion surveys!
In a very basic sense a poll is just the accumulated data of asking people the same question about their opinion.
Note - there’s nothing there about good methodology, or checking that the question is unbiased, or even how you choose who you talk to. Yet the word poll itself has an almost magic quality, especially when shared in the media. Poll in that universe means a scientific way to actually try to predict something about the opinons of a larger group of people.
Now there are research firms that do hold to the latter definition, but it’s this discrepancy between what a poll is in a basic sense and its ideal form that is often exploited for agendas.
This is how you get a journalist having a bit of fun on Twitter transformed into headlines that read “Jeremy Corbyn tops poll for best prime minister Britain never had” and a (quite absurd) integration by Kay Burley of Toby Perkins MP, with the MP rightfully pointing out that it doesn’t have merit in actual opinion polling.
To put it in more absurd terms, sometimes something like this:
Becomes a headline like this:
Of course, not every polling pitfall is as clear as that but this newsletter is designed so that with time you’ll be able to figure out beyond the headline whether a statement has any worth.
What is a good rule of thumb for thinking critically about polls?
A good poll selects a suitable method to answer a specific question (or set of questions), and makes sure that it speaks to a representative selection of people from the group whose opinion it would like to predict.
How polls can project opinion from a 1000 person sample to a whole country has been famously described by Gallup as like checking if soup needs to be salted - you don’t have to try the whole bowl, just make sure the soup is mixed enough that your tablespoon reflects its actual taste.
With that in mind a good way of quickly checking whether a poll might be not as accurate as it claims is going through the five Ws you were taught as a child (and the one How):
Who did they ask the questions to?
What order were the questions asked in?
When were the dates they asked these questions?
Where were the people asked?
Why are the question being asked?
How did they do it?
Not all of these questions will be relevant to every poll, and not all of the answers will give a red flag. If we take the above Dr Rob Robinson poll and do a sense-check on it we find the below answers:
Who? Random people on Twitter, most of whom probably follow Dr Rob Robinson.
What? Not relevant.
When? August 31st.
Where? Could be anywhere in the world Twitter is available.
Why? To see if people prefer pineapples or mangos.
How? Asked people on Twitter and anyone could choose to answer if they wanted - also was suggesting people vote for mango.
Just doing that alone, without any extra knowledge, we can see that this is not really something to be taken seriously.
Additionally we can see clearly that the original headline is twisting the results, saying that Dr Rob Robinson did a 12,000+ people survey on the nation’s favourite fruit, when really he was asking whether his followers (who could be anywhere in the world) preferred pineapples or mangos.
A lot of polling companies provide tables that can make answering these questions easier. In the UK these are a bit more straightforward to access than a lot of other countries as the British Polling Council (a voluntary association of market research companies that produce opinion polls) requires its members to publish tables within two working days of their results being shared in public. However most reputable polling agencies will have an Archives page where you can download an excel file or equivalent that clearly details the methodology.
Next week we’ll be looking in greater depth at polling data tables and their compenent pieces!
As always, if you'd like to drop me a note, you can contact me by replying to this email or over on Twitter at @thesecondrussia.