I mentioned this research, by Ashok Kaul and Michael Wolf, last year. Although The Observer says the working papers were "widely disseminated in the media", I don't recall them getting much coverage at all and they were conspicuously ignored by Cyril Chantler when he conducted his review.
Pascal Diethelm, an anti-smoking campaigner who unveiled a not-very-scientific graph when I saw him speak in November, claims that the research contains seven errors. “Taken individually," he says, "most of them are sufficient to invalidate the findings of the papers. Collectively, they are damning.”
Kaul and Wolf have hit back strongly at this allegation with two rebuttals. In the second and most detailed of these, they conclude:
The authors of the annex have set out to discredit our research by providing a list of seven (so-called) errors and a list of seven issues. But they have clearly failed in their mission. Although there are some (minor) points of debate, there is not a single “extremely serious” error in our two working papers, as we have explained in detail in this reply.
Instead, the authors of the annex (i) have shown a surprising lack of basic statistical knowledge and (ii) have made false statements about the content of our working papers. What they have achieved to discredit, therefore, is only themselves. Perhaps this serves to explain the anonymous nature of the annex.
We welcome a constructive debate based on substance and comporting to scientific stan- dards and decorum. We regret that the authors of the annex have repeatedly overstepped the limits of scientific debate (i) by misrepresenting our approaches, methods, and find- ings; (ii) by engaging in personal attacks; and (iii) by hiding behind the cloak of anonymity. Although we firmly believe that constructive criticism will further scientific discovery and support good policy-making, we cannot accept defamatory statements and unsubstantiated attacks both on our academic institutions and on us as individuals.
You can read the whole back-and-forth here if you have the knowledge and inclination. Most of the issues are of a technical nature and go over my head, but the first of the seven supposed 'errors' is straightforward and it seems clear to me that it is not an error at all, let alone one that "invalidates the findings of the papers".
Kaul and Wolf's anonymous critics argue that absence of evidence is not conclusive evidence of absence. They complain that PMI issued a press release which said that 'The plain packaging experiment in Australia has not deterred young smokers, professors from the Department of Economics at Zurich University and the University of Saarland found in a report released today'. They also complain that BAT said the studies show that 'there has been no change in the pre-existing trend in youth or adult smoking since the introduction of plain packaging'.
They argue that Kaul and Wolf did not prove conclusively that there was no effect on smoking rates, only that their studies failed to find an effect. This might seem a pedantic distinction—in casual conversation, this is exactly how we might refer to such evidence—but it is a fair point. However, Kaul and Wolf have always been careful to make this distinction and have never oversold the findings. Their studies are very cautiously worded and they never ruled out the possibility that plain packaging had an effect that were were unable to measure.
The complaint of Kaul and Wolf's critics is essentially that other people drew a firmer conclusion from their research, but that is an absurd justification for retraction. I don't recall any demands for retraction when this study was reported with headlines such as 'Australia’s plain cigarette packaging has not given a boost to the illicit tobacco trade' and 'Cigarette plain packaging fear campaign unfounded, Victoria study finds'.
To take a more recent example, the BMJ published a study on Tuesday that was widely reported as showing that 'Alcohol has no health benefits after all' (The Times). This is not what the authors of the study said, nor is it what the study showed, but nobody would suggest it should be retracted because of inaccurate reporting by third parties. If that were the criteria, the public health literature would be very thin indeed.
I'd be interested to read comments from any readers who have the statistical qualifications to make a judgement on the other points, but if the rest of the critique is of the same standard as this I doubt Kaul and Wolf have anything to worry about. The intention seems to be to throw a little mud so that readers of The Observer—which has never mentioned the studies until now—get the impression that they have been debunked.
