What's in a question?

I missed this when it came out in November, but it's a very interesting example of how subtle changes in wording can yield different survey results. Last year it was reported that a relative majority of smokers now supports plain packaging in Australia. The survey was published as a study in Tobacco Control and these were the results:

Support for PP [plain packaging] increased significantly after implementation (28.2% pre vs 49% post), such that post-PP more smokers were supportive than opposed (49% vs 34.7%).

Although it seems trivial, there was a subtle shift in survey methodology after plain packaging came in. The Tobacco Control researchers asked a slightly different question to smokers after plain packaging was in force.

Before implementation the smokers were asked:

‘Tobacco companies should be required to sell cigarettes in plain packages—that is, in packs without the usual brand colours and symbols, but keeping the warning labels’

After implementation, they were asked:

‘Tobacco companies should continue to be required to sell cigarettes in plain packages, as they are now’.

The difference seems insignificant at first glance and, given the context, the rewording seems reasonable. But the wording of the questions had more of an effect than you might think. Shortly after the Tobacco Control paper was published, the research agency Povaddo conducted a survey in which 500 smokers were asked the first question and 500 smokers were asked the latter question.

Of those who were asked the first question (‘Tobacco companies should be required to sell cigarettes in plain packages—that is, in packs without the usual brand colours and symbols, but keeping the warning labels’) 24 per cent agreed and 46 per cent disagreed (the rest were indifferent).

Of those who were asked the second question (‘Tobacco companies should continue to be required to sell cigarettes in plain packages, as they are now’) 31 per cent agreed and 36 per cent disagreed (the rest were indifferent).

This is quite a swing and it is interesting to speculate on why opposition to plain packs eased off by ten percentage points when respondents were referred to packs 'as they are now'. The most likely explanation is status quo bias, possibly combined with the bandwagon effect.

It seems that the wording of the question in the Tobacco Control study was more important than it appeared. Even without any change in smokers' attitudes towards plain packaging, the wording ensured that they would appear to be less hostile to the policy.

But there is another big difference between the two surveys which cannot be explained by the wording of the questions. In the Povaddo survey, a relative majority of both sets of smokers were against plain packaging. The most it could manage, using the second question, was 31 per cent in favour and 36 per cent against (as opposed to Tobacco Control's 49 per cent in favour and 35 per cent against).

How can this be explained? The Tobacco Control survey was conducted in mid-2013 and it is possible that smokers' attitudes have hardened in the meantime. Possible, but not very likely. A better explanation lies in the other questions that the Tobacco Control survey asked, but the Povaddo survey didn't. These additional questions are listed below:

Desire to quit (‘how much do you want to quit?’: not at all, a little, somewhat, a lot)

Quitting history (none versus at least one quit attempt: since the last survey/in the last year- for new recruits), with being currently quit at the time of the interview constituting a quit attempt. ‘When did your most recent (or current) quit attempt start?’

Quitting self-efficacy (‘If you decided to give up smoking completely in the next 6 months, how sure are you that you would succeed?’ : not at all sure, slightly sure, moderately sure, very sure, extremely sure)

Proximity of intention to quit (no intention or beyond 6 months/within the next 6 months).

Perceived risk of disease (‘if you continue to smoke as much as you do now, what are the chances that you will get a smoking-related disease, such as lung cancer, heart disease, or emphysema?’ : very high, somewhat high, neither high or low, somewhat low, very low);

Realised health damage (‘To what extent, if at all, has smoking damaged your health?’ : not at all, just a little, a fair amount, a great deal).

You will have noticed that all these questions refer either to quitting smoking or to the negative health effects of smoking, thereby framing the question about plain packaging in a context in which any tobacco control policy might seem more reasonable.

It is not clear from the text of the Tobacco Control study whether these questions were asked before or after the plain packaging question, but if they were asked first—and I'll bet you a pig to a pork scratching that they were—then this framing, combined with the changed wording of the post-implementation question, explains why the Tobacco Control researchers got a response that was more to their liking than the more neutral Povaddo survey.

Share

Tweet