18 May 2015
Remember the recent UK elections? The ones where all the pollsters utterly failed to predict the majority win by the Conservative Party? Even though some of the world’s largest research groups were undertaking polls? It left a fair bit of egg on the idea that we can actually predict anything. And it was also an almost-perfect case study of how traditional sampling methods fall short.
So what went wrong?
The problem starts with non-response…
People opting not to respond to surveys is not a new problem, but rather a growing problem – and it’s been swept under the rug for a while now.
Consumer attitudes towards surveys are changing – with time pressure, security concerns, and the ease of opting out of self-complete questionnaires all playing a part. This shift has resulted in interviewers experiencing higher refusal rates than ever before.
Another big contributor to this is the technology-driven era that we are living in. When it’s just you and a survey on your phone, or on your computer, it’s that much easier to opt out if it’s too long, too boring, or there’s simply not enough of an incentive to complete it.
…and is compounded by coverage bias
The real problem comes in with the fact that non-response exacerbates coverage bias.
In digital surveys the problem of coverage bias means that because each sample platform has its own inherent skews (demographically or otherwise), you can’t reach a representative sample of the entire population through one sample group.
The increase in non-response compounds coverage bias in both digital and pen-and-paper surveys, because even fewer people within the sample group are responding, resulting in even further skews. This makes it extremely difficult, if not impossible, to achieve a truly representative sample.
How do we test the validity of our sampling frames?
The best way we’ve found of testing whether a given sampling frame yields accurate and reliable data is to compare its data with reliable external sources of data (e.g. AMPS or the IEC in South Africa) to confirm its validity, and then repeat the same study to ensure stability and reliability.
This can then be repeated with different combinations of sample bases.
Investigating a sample base is a little like kicking the tires on a new car – you need to be sure of what you’re getting before you take it out on the road.
So what can researchers do to improve their sampling methods?
1. Boost response rates
Shorter questionnaires, a gamified approach to research design, and experimenting with different ways of incentivising respondents can all help in reducing non-response.
2. Use more respondent bases
In order to reduce coverage bias, Pondering Panda is now sampling across multiple bases. With each base – whether it’s a social media platform, online community or panel, mobile community or mobile list provider – having its own skews, we sample across multiple bases as standard for any nationally representative study. If a very specific target group is of interest, clients need to select the base (or combination of bases) that is most representative of that demographic target group.
While this doesn’t eliminate the problem entirely, it does mitigate it by balancing out the biases inherent in individual bases or groups.
Meanwhile, back in Britain…
All this brings us back to the recent UK elections. Why did the pollsters fail, and how could this have been avoided?
Since the elections, Nate Silver (a guru in predictive analytics and forecasting) has been quoted in many articles as having argued that the pollsters failed because of their inability to achieve representative samples, which in turn, led to invalid data.
He also mentioned this is an increasing problem worldwide, which points to the problem of increasing refusal rates to participate in surveys, as well as coverage bias (due to these polls being conducted online). Another contributing factor to the flawed prediction was that many believed people didn’t want to indicate to pollsters that they were going to cast a ‘politically incorrect’ vote, but that’s another can of worms entirely.
If you’re interested in predictive research, we would heartily recommend Silver’s book, ‘The Signal and the Noise: The Art and Science of Prediction’.
Questions or comments? Leave them below…