Multi-market data quality: where cross-country comparisons go wrong

Every multi-country study produces differences between markets. Some of them are real, and some are the instrument, the mode or the sample frame showing through. Analysis cannot reliably separate the two after the fact, so the separation has to be designed in.

Scale use is not universal

Response style differs systematically by culture. Some markets use the extremes of a scale readily and others avoid them, and the effect is large enough to reverse a ranking between countries. Where cross-market comparison is the point of the study, this needs either standardisation within respondent or a scale design less sensitive to style.

Mode differences masquerade as market differences

A study fielded online in six markets and face to face in three has a mode effect sitting exactly on top of its market comparison. If the design cannot avoid mixed modes, the overlap needed to estimate the effect has to be built in deliberately — a subsample fielded both ways in at least one market.

Translation shifts meaning quietly

• Frequency words (often, regularly, sometimes) rarely translate to equivalent intervals
• Brand and category terms may be generic in one market and proprietary in another
• Politeness conventions affect how a negative answer is expressed
• Agreement scales carry different social weight across languages

What to do about it

Back-translate, then have a researcher who works in the market read the instrument for meaning rather than accuracy. Pilot in the two markets most likely to diverge. And report the known limits of comparability alongside the comparison, rather than leaving the reader to assume there are none.