At the end of last month, I revealed the e-mails that show Public Health England telling the Sheffield Alcohol Research Group to change the methodology of their computer model despite the obvious reservations of the latter. The change in methodology had no sound scientific justification but it had the effect of lowering the model's implied drinking guidelines for men from around 21 units to around 13 units. The Chief Medical Officer subsequently announced that the male drinking guidelines would be reduced from 21 units to 14 units.
If you haven't read my article, please do so before you go any further.
Public Health England's Duncan Selbie has since written a letter to the
Spectator in which he disclaims responsibility for this and gives the credit/blame to the guidelines group. I reproduce it below, along with my response.
Christopher Snowdon’s piece, ‘The new drinking guidelines are based on massaged evidence’, is grossly incorrect and misrepresents Public Health England’s (PHE) role in the guidelines’ development.
PHE emphatically refutes any suggestion that we intervened in some way to influence the evidence made available by Sheffield University to an independent expert group, the Guideline Development Group (GDG), which was set up by the UK Chief Medical Officers to help develop the alcohol guidelines.
As part of the secretariat to the group, we commissioned the analysis, as requested by the GDG, from Sheffield University. Any emails from PHE to Sheffield commissioning additional modelling and evidence were based on the GDG’s decisions and at their request, as is clearly shown by the publicly available minutes of their meetings.
This has been confirmed by Sheffield University’s Alcohol Research Group, which has said:
“Minutes from the subsequent GDG meeting on 21 January 2015 state that, after hearing Sheffield’s presentation of their work, the GDG concluded: ‘A holistic, expert judgement on guideline levels would be needed, taking account of uncertainties and issues not fully modelled’. This demonstrates that the group recognised there was considerable scientific uncertainty present and that no single piece of evidence or modelling decision used in isolation would determine the final guideline.
“As noted in the Royal Statistical Society’s consultation response: “This is a contested area of science with considerable uncertainties” (paragraph 1.1). The change to the base case analyses related to a point of scientific uncertainty. The Sheffield Alcohol Research Group were happy with the decision taken whereby the base case analysis was revised but the original modelling assumptions were retained as one of a series of sensitivity analyses.
“Those analyses explored major areas of uncertainty within the underlying evidence and their implications for the Guideline Development Group’s work. The group considered those sensitivity analyses in detail and took them into account in their decision-making.”
Mr Snowden [sic] also refers to The Public Health Burden of Alcohol and the Effectiveness and Cost-Effectiveness of Alcohol Control Policies: An evidence review, which PHE published in 2016. The facts speak for themselves; this unprecedented and comprehensive evaluation of the evidence had an extensive three-stage peer-review involving UK and international academics. The abridged version was also subject to further peer-review processes before its publication by The Lancet.
My reply:
If, as Public Health England claims,
my exposé of the alcohol guidelines review was 'grossly inaccurate', one would expect the agency to flag up some factual errors. Their sole complaint appears to be that I suggested that it was they, rather than the Guideline Development Group, who came up with the idea of changing the methodology. I suggested no such thing. I explicitly said in the original article:
"On 21 January, the GDG held a meeting at which SARG’s John Holmes and Colin Angus presented their findings. The
minutes of this meeting contain the first mention of an idea that would
have a profound impact on the whole project. It was suggested that SARG
researchers should ‘estimate risk curves without threshold effects for
wholly alcohol-attributable chronic conditions’."
As the commissioner of the research, PHE followed up with a series of e-mails to the Sheffield Alcohol Research Group (SARG) in February 2015. These e-mails show the Sheffield researchers trying to talk PHE out of the idea of replacing their base case scenario with a sensitivity analysis that was built on patently unrealistic assumptions.
The crucial e-mail exchange came on 10 February when SARG said that they were 'unclear exactly what was being requested' and that 'it does not seem right to assign people drinking at very low levels a risk of acquiring alcoholic liver disease and similar conditions.' They added that: 'Unless there are strong opposing views, we think it better to keep the threshold in the base case.'
That e-mail was sent at 4.37pm. PHE replied at 10.40pm with a short
e-mail ignoring the compromise suggested by the Sheffield team and
asking how much it would cost to drop the threshold from their main
findings (the 'base case'). From this, there was no looking back.
It is also possible that the GDG had explicitly ordered the base case to be changed in its meeting of 21 January. If so, it was not recorded in the minutes. The minutes merely state that the GDG had suggested that PHE 'commission further sensitivity analyses or new modelling where feasible' including an estimate of 'risk curves without threshold effects for wholly alcohol-attributable chronic conditions'. As I explained in my article, there is a huge difference between running a sensitivity analysis and changing the base case that will dictate the headline findings.
The reaction of the Sheffield researchers to PHE's request for a change in methodology suggests that such a fundamental change was not agreed at the GDG meeting. Two of the Sheffield team were at that meeting. If the GDG had explicitly called for the base case to be changed, why were they so surprised and dismayed when PHE asked them to do it?
We will probably never know what was going on between PHE and members of the GDG behind the scenes. PHE's complaint against me seems to be that I gave the impression that the agency somehow overrode the will of the GDG. I don't think any fair-minded reader would infer that from my article but, for the avoidance of doubt, let me say again that the GDG was packed with campaigners from the temperance and 'public health' lobby who would have been delighted to see the guidelines lowered. There is enough blame and shame to be shared by all three of the groups involved: the guidelines committee, Public Health England and the Sheffield Alcohol Research Group. I have never suggested that PHE bear all, of even most, of the responsibility.
The fact remains, however, that Public Health England commissioned and funded the Sheffield research and it was they who set the parameters, whether acting as intermediaries or on their own initiative. Having funded the research, they now seem keen to pass the buck to the GDG. For their part, the Sheffield team has always been keen to pass the buck to PHE. Twice in their report, SARG stress that the dropping of thresholds was done at the agency's request. On page 28 they say: 'At the request of the commissioners (Public Health England), this threshold effect removed for the base case analysis...' Note that they do not say that it was done at the request of the GDG. Elsewhere in the report they make it clear that they would not normally run their model in the way they did and, at times, they seem to be distancing themselves from their own findings.
If this research was so strong, why does nobody want to take responsibility for it? Why is PHE so keen to point the finger at the GDG? Why is SARG so keen to play down the importance of their report? It is not a good look for PHE to be saying that they were only following orders. The best defence the Sheffield team has been able to mount is to claim that the change in methodology related to 'a point of scientific uncertainty' and that they were 'happy with the decision taken'. Anyone who reads the e-mails can be the judge of how 'happy' they were about it, but there is really no 'scientific uncertainty' about whether moderate drinkers are at an increased risk of developing diseases of alcoholism. They are not. There is plenty of uncertainty about how much an individual needs to drink before the benefits of alcohol are outweighed by the risks, but there is no uncertainty about there being a threshold for the ten chronic diseases that SARG dropped 'at the request of the commissioners'.
SARG are keen to point out that the change in the guidelines was not due to any single piece of evidence. I have never suggested otherwise, but their report cannot be dismissed as unimportant. It was explicitly designed 'to inform the Chief Medical Officer's review of the UK low risk drinking guidelines', as its title says. The first draft noted that the 'implied guideline thresholds are generally similar to those in the current UK lower drinking guidelines’. An anonymous reviewer of that draft wrote: 'I predict that there will be very little, if any, change to the Guidelines'. That all changed once PHE got SARG to change their methodology. It would have been possible to lower the guidelines without changing the Sheffield model, but it would have been a hard sell. The credibility of Sheffield's work is therefore of significant public interest. Is there anybody willing to stand up for it?
PHE defends its 2016 alcohol policy report on the basis that it was peer-reviewed. Peer review can make a document fit for publication but it does not make it true. If the aim of peer review is to fact-check, PHE was let down by its reviewers on that occasion.
The 2016 report contains numerous basic errors, such as the claim that ‘real-term alcohol prices have decreased’ since 1980 (they have risen by 23 per cent). The report was released to the public with the claim that people in Britain are drinking twice as much as we did in 1980. We are actually drinking exactly the same amount (9.4 litres per adult). 'The facts speak for themselves', says Mr Selbie. Indeed they do, if you can find them.
When a report is littered with obvious errors, defending it on the basis that it has been peer reviewed is more of an indictment of your peer review process than a defence of the publication. But I am glad they have brought it up since it allows me to mention something about the guidelines process that space did not permit in my original article.
When I have had work peer reviewed at the Institute of Economic Affairs and elsewhere, the editor gives the manuscript to a third party of his or her choosing who doesn't know that I wrote it. When I receive their comments I don't know who they are. This double-blind system prevents any bias towards or against the author. By convention, reviewers are not paid.
This is not how it worked when the Sheffield team submitted their final report. On 6 May 2015, PHE sent an e-mail asking if they had had it externally peer reviewed. Sheffield said that they hadn't but they were 'happy to arrange for this to happen'. On 12 June - more than a month later - someone from PHE replied to say that 'I'm happy for you to suggest reviewers'. Sheffield then provided two names (which are redacted in the e-mails released under FOI) and asked PHE: 'Do you want the reviews to come through you or are you happy for us to just share the comments and revisions?' PHE replied: 'Happy for you to sort out the peer review directly and share comments/revisions with me.'
On 25 June, Sheffield e-mailed PHE saying that they had approached two potential reviewers, one of whom wanted £650 to do it because 'this is the rate he is being paid by PHE to review another lengthy report'. PHE said that they 'don't have any funds set aside for peer review' but were 'looking into' whether they could access some.
By mid-September, no funds had been released and only one reviewer had been found - an unnamed PHE employee. SARG e-mailed to say that 'I think we're just looking at XXX(PHE) unless you would like me to try to find someone else at short notice?' It is not clear from the released e-mails whether a second person was ever found. All we know for sure is that PHE handed the task of finding reviewers for the SARG report to the people at SARG who then approached somebody at PHE to do it! This is a far cry from blind external peer review.
Finally, Mr Selbie quotes SARG quoting the Royal Statistical Society and their reference to a 'contested area of science'. SARG say that the 'change to the base case analyses related to a point of scientific uncertainty'. This is misleading. The contested area of science to which the Royal Statistical Society referred was the general issue of defining a low-risk guideline, which is indeed difficult to pinpoint with precision. They were not referring to the specific question of whether there is a consumption threshold for some diseases.
SARG did not quote what the Royal Statistical Society actually said about that question. On page two of
their consultation response, the Society notes that getting rid of the threshold leads to a 'statistically implausible
assumption of a linear relationship' and that 'without this enforced assumption, the threshold for males to reach a 1% lifetime risk would have been 21
rather than 14 units, exactly the previous Guideline'.
