Survey respondents: finding the right people and avoiding the obvious biases

The interesting failures in sampling get written up. The common ones do not, because they are dull and everybody has made them. In our experience four mistakes account for most of the damage, and all four are visible in the brief.

1. The audience is defined by who is easy to find

A target definition that maps neatly onto standard panel profiling variables is often a definition that was shaped by them. It is worth asking whether the audience you wrote down is the audience you care about, or the nearest one that could be sampled quickly last time.

2. The mode decides the sample

Choosing online because it is fast and then weighting to census targets does not recover the people who were never reachable. Weighting adjusts the balance of who you got. It cannot conjure the households you never contacted.

3. The screener teaches the respondent what to say

A screener that asks whether someone is a decision-maker, in a survey that pays, will find decision-makers. Screeners that establish qualification indirectly — through behaviour, through knowledge, through questions whose qualifying answer is not obvious — get closer to the truth.

4. Incidence is estimated once and never revisited

An incidence assumption made at proposal and never checked is the single most common cause of a study going sideways mid-field. Funnel data exists from the first day of fielding. Looking at it on day one, rather than at the end of week two, converts a crisis into a decision.

What to do instead

• Write the audience definition before looking at what is easy to sample
• Choose mode from the population, then price it
• Qualify on behaviour and knowledge, not on self-description
• Read the screening funnel on day one and again on day three
• Ask your supplier what they would change about the definition — and listen to the answer