Tricks of the trade

Over the last few years I have spent an inordinate amount of time picking through 'heart miracle'  studies - ie. those which claim a dramatic fall in the heart attack rate as a result of a smoking ban. What I find fascinating about them is that you know, from the very offset, that the conclusion it going to be wrong. 

It is not a question of having a closed mind or being unduly suspicious. Smoking bans cannot bring about huge reductions in the heart attack rate of 10, 20 or 30 per cent because it is a mathematical impossibility. Only someone who does not understand the difference between absolute risk and relative risk would think otherwise (I have tried to explain this before, and will do so again on this blog soon).

And so it is like watching David Copperfield doing his flying trick. It might look impressive but you know it's just a trick. With magic, the fun is in trying to work out how it's done. And just as the flying trick is always going to involve some form of industrial strength wire, the explanation to a heart attack miracle is always going to involve some form of cherry-picking.

In America, there is no national smoking ban. Instead, individual towns often bring in their own smokefree legislation. This provides a perfect hunting ground for the cherry-picker because hundreds of individual bans mean hundreds of chances to find an unusual drop in hospital admissions. As Michael McFadden points out in the comments to this post, who knows how many places they examine before they find a place which gives the "right" result?

One of the great mysteries of antismoking research that we'll probably never know the true data for is just how many studies like this have been started and then aborted when the early answers didn't come out right.

The smaller the community, the greater the chances of large random variations. It is no great surprise, then, that the first heart miracle was seen in the small town of Helena, Montana (population 66,000). The 40% drop seen there was possible because there were so few people being hospitalised (as few as 4 per month), therefore a small variation in actual cases created a huge variation in the percentage.

But as the size of the study populations grows, the opportunity for spotting big variations in the heart attack rate declines. This can be seen if you compare the supposed drop in heart attacks to the size of the study populations:

Bowling Green, Ohio (30,000) : 47%

Helena, Montana (66,000) : 40%

Pueblo, Colorado (105,000) : 33%

Piedmont, Italy (4.4 million) : 11%

Scotland (5 million) : 17%

England (51 million) : 10%

There is, of course, no reason why a smoking ban should be more effective in preventing heart attacks in Helena, Montana than in the whole of England, so right away we have reason to be suspicious. And it is when whole nations are studied that the heart miracle theory really falls apart. 

There are two key reasons why a cherry-picker is going to find it difficult to use British data to sustain the heart miracle hypothesis. Firstly, the UK's population is too large to ever see random variations of 30 or 40%. Secondly, the NHS publishes hospital admissions data, so we can easily check what we are being told.

This was the problem for Jill Pell and her Scottish study. No sooner had she claimed a 17% fall in heart attacks than the NHS published hospital admissions data that showed that the real decline was less than half of that.

Anna Gilmore is the person charged with showing that England saw a heart miracle after the ban. Her position is even worse than Pell's. Not only has England got a much larger population than Scotland (and therefore even less chance of a fluke result), but the hospital admissions data has already been published. 

We know that the heart attack rate fell by between 2 and 4% before the ban and by between 2 and 4% after the ban (see the HES website). To date, we only have the data for the first 9 months following the smoking ban, but that it is enough to go on. After all, if smoking bans immediately save lives, the first 9 months is where we would see the biggest drop.

If the claims being made in recent news reports are to be vindicated, Gilmore needs to turn this 2 to 4% drop into a 10% drop. It sounds a tall order but it is not impossible.

The obvious way of conducting this study would be to use the NHS's hospital admissions data from July 2007 to June 2009 and compare the number of heart attack admissions in that period to the two years prior to the smoking ban. However, that is unlikely to yield a 10% fall, so a more creative approach may be required. 

Here are some tricks she could use...

Possibility no. 1

Pick an unrepresentative time-frame

There will some degree of seasonal variation in the hospital admissions figures. If, for example, there was an unusual decline in admissions between October and December 2008 compared with October to December 2006, Gilmore could focus entirely on those months and ignore the rest.

Tried and tested in: Wales, where Chris Franks ignored inconvenient months to 'prove' there was heart miracle. And in France, where a single month was used to show the same.

Possibility no. 2

Change the definition of a heart attack

'Heart attack' is a general term, usually equated with the medical condition of acute myocardial infarction (AMI). This is what the Helena researchers studied and it is what the Scottish researchers initially set out to study. Somewhere along the way, however, the Scottish researchers decided to focus on acute coronary syndrome (ACS). Nothing wrong with that per se, since ACS can also be reasonably classed as a heart attack. 

But there is a difference between AMI and ACS, and there are different figures for the two. Gilmore could decide to use whichever set of figures shows the biggest drop. She could even do what Jill Pell did, and use the troponin test to diagnose ACS cases. This would have the advantage of meaning that no-one would ever be able to disprove her study, since the NHS does not use the troponin test to diagnose ACS cases.

Tried and tested in: Scotland, by Dr Jill Pell and her team

Possibility no. 3

Pretend the smoking  ban started in a different year

If the data for 2007 and 2008 doesn't fit, why not make out the smoking ban started in a different year?

Tried and tested in: Bowling Green, Ohio, where -  incredibly - researchers used a drop in heart attacks between 2002 and 2003 to 'prove' there was a heart attack miracle. 2002 did indeed see an unusual peak in heart attacks, and the rate fell in 2003. This was taken as proof that the smoking ban "slashed" heart attacks. The inconvenient truth was that the smoking ban actually started in early 2002.

Possibility no. 4

Ignore the whole population 

This is a variation on No. 1, but with the region being cherry-picked rather than the time-frame. There is bound to be regional variation in the hospital data, so why not concentrate on the areas with the biggest heart attack decline and ignore the rest? 

Tried and tested in: Piedmont, Italy, and by the Daily Mail who reported that the heart attack rate had fallen by 40% in "some hospitals" in England, based on data from the Shrewsbury and Telford Hospitals NHS Trust which saw a large but unrepresentative fall in cases. 

Of the latter case, even ASH felt compelled to add a disclaimer when it reported the Mail's story:

Please note: Although it was reported by some newspapers that hospitals saw a 40% decline in heart attacks the overall figure was 3%.

Having made that clear, it will be interesting to see how ASH responds to the 10% claim.

I confess that I know very little about Anna Gilmore, other than that she is a Board & Council member of ASH and has worked with Stanton Glantz. I'm sure she is not the type of person who would use any of the tricks above.

I'm just saying that if an unscrupulous researcher did want to manipulate the evidence for political ends, these are the kind of tricks they could use. That's all I'm saying.

Incidentally, Dr Michael Siegel has, as usual, some interesting things to say about the 10% claim. He concludes his analysis by saying:

The scientific integrity of the tobacco control movement is imploding before my very eyes.

Do go read the rest.