Who will check the fact-checkers?

In the post-truth world in which we supposedly live, the rise of professional fact-checkers is welcome. The BBC's More or Less is the pick of the bunch, but it's only 30 minutes on Radio 4 for a few months each year. Online, Full Fact does a great job of picking apart the numbers of the day. 

Channel 4's FactCheck is less impressive but it can be useful when it isn't buckling to political activists (as it did last week when it changed the headline and website address of this article).

And then there is the BBC's Reality Check which is sort of a fact-checker, but also sort of a rough guide to the issues people are talking about. Its premise seems to be that it provides a cool-headed look at The Facts which you wouldn't get from the rest of the media (like, er, BBC News). Last night it put out an article which illustrates the potential for abuse when journalists set themselves up as arbiters of The Truth.

Written anonymously, the article is titled 'Reality Check: Why ban fast food within 400m of schools?'and is about Sadiq Khan's idiotic plan to protect chicken shops from competition tackle obesity.

What does the research show? There's quite a lot of evidence that having fast food nearby leads to more obesity in adults.

I happen to be familiar with the research in this area. There is indeed a lot of evidence, and very little of it makes a convincing case that living near a fast food shop leads to more obesity in children or adults. But there is one particular UK study that campaigners often cite because it claims to have found a link - and that's the one that BBC Reality Check focuses on: 

There is, for example, this research from Cambridge, which found that people living closest to the largest number of takeaway food outlets were more than twice as likely to be obese than of normal weight.

This study concluded that: 'Exposure to takeaway food outlets in home, work, and commuting environments combined was associated with marginally higher consumption of takeaway food, greater body mass index, and greater odds of obesity.' But there is something rather interesting about it that was picked up by the statistician Jeremy Franklin. He noticed that:

After reading the interesting article by Burgoine et al. I was at first irritated by the lack of a table to compare the characteristics (as shown in Table 1) of participants grouped according to quarters of take-away environment... Usually one would expect such tables in order to assess the comparability of the groups with respect to possible confounders and for a direct, unadjusted comparison of outcomes, respectively.

Then I discovered this information in Web table 3 of the online appendix. Here, we see systematic differences between quarters with respect to education, smoking and car ownership...

What surprised me even more in Web table 3 was the fact that mean take-away consumption was slightly inversely correlated with combined take-away availability, varying between 36.3 g/day in Q1 and 34.2 g/day in Q4. This contrasts completely with the results of the multivariate analysis (Fig. 1) in which a significant positive correlation between take-away availability and consumption was obtained. Moreover, In Web table 3 mean BMI is almost constant in all quarters of take-away availability, contrasting with the significant positive correlation between take-away availability and BMI derived from the multiple linear model (Fig. 2).

In other words, there was no difference in obesity rates between those who lived near fast food outlets and those who didn't. Moreover, the people who lived near them actually consumed slightly less takeaway food. Here's the data that was tucked away in a supplementary file:

The findings presented by the authors are entirely the result of changes they made to the data in their attempt to control for other variables.

Perhaps some of these adjustments were appropriate, but we are required to put a lot of faith in the researchers before we accept their conclusion. And their conclusion is really that obesity rates are not actually higher near takeaways but they would be were it not for confounding factors.

A further point of interest is that the study included supermarkets as a source of takeaway food. When the authors excluded supermarkets in their sensitivity analysis, they were unable to find any association with between takeaway food and obesity, even after adjusting the data. They admitted that excluding supermarkets meant that 'the associations between combined take-away food outlet exposure, consumption of take-away food and body mass index were attenuated towards the null’.

As Franklin says, 'The expression "attenuated towards the null" is an understatement: no association remains at all, in agreement with the simple univariate comparison.'  

The Cambridge study is therefore hardly the most compelling evidence that having fast food nearby leads to more obesity.

But the Reality Checkers have two other pieces of evidence to make the case. The first is a survey conducted by Brent council, asking secondary school pupils how far they would be prepared to walk to a fast food shop. It didn't included any measure of obesity or health and, as the BBC acknowledges, 'the differences between the likelihood of children having lunch from a takeaway outlet if they attend a school close to one or further away from one were fairly small and the results were skewed by most of the children surveyed not actually being allowed off-site at lunchtimes.'

Finally, there is this:

Another report on the subject found that the food available near schools did have some effect on pupils' choices but that it was only a small effect.

That's putting it mildly. The association between proximity to takeaways and an 'unhealthy diet' was tiny (0.003, 95%CI 0.001 – 0.006) and the study didn't even attempt to find an association with obesity.

So much for there being 'quite a lot of evidence that having fast food nearby leads to more obesity'. The most revealing thing about the BBC article is not how weak the evidence it cites is, but the failure to mention all the other evidence.

For example, this study found that 'obesity prevalence was highly significantly negatively related to the densities of both FFRs [fast food restaurants] and FSRs [full service restaurants]' and this study found that 'away from home food expenditures negatively affect BMI and that BMI is negatively related to the percentage of the food budget spent away from home'.

This study from the UK found that fast food consumption was negatively associated with obesity (ie. those who eat it most often have the lowest body mass). Although the authors made significant adjustments to the data, they were not able to find a positive association. The raw data is shown below.

This US study concluded that 'Proximity of "fast food" restaurants to home or work was not associated with eating at "fast food" restaurants or with BMI' and this US study found 'no association between child overweight and proximity to playgrounds, proximity to fast food restaurants, or level of neighborhood crime.'

And this study from the UK didn't find an association between takeaway outlet density and obesity except among 'the least educated'.

The evidence that living near a takeaway (or near lots of takeaways) is not at all strong. It is mixed and conflicting, with many results supporting the null hypothesis. In their literature review of 2010, Fraser et al. found that...

... of the 12 cross-sectional studies which looked at FF [fast food] outlets in relation to overweight or obesity, six found a significant positive association, two had significant negative results and five showed no association. Of the studies which showed a positive association between FF outlets and weight/BMI, one only found an association in non-car owners, one found an association in adult females only, one found a significant association between increased number of FF outlets and increased obesity but also decreased obesity if closer to a FF outlet, and one found an association between weight status and FF exposure in schools. The other study with a positive result aggregated their individual level data to perform a county level analysis. All six of these studies used self-reported heights and weights to calculate BMI. The longitudinal study found no association between density of FF outlets and BMI change in children.

Faced with this murky picture, BBC Reality Check chooses to simply assert that there is 'quite a lot of evidence that having fast food nearby leads to more obesity in adults' and the quotes Prof Naveed Sattar from Glasgow University who offers the kind of opinion you'd get from a bloke in a pub.

"When I was a child we had a fish and chip shop about 200m from my school and lots of us went there - if it had been a bit further away maybe they wouldn't have bothered.

"It's pretty obvious that if you make things easy people will gravitate towards them."

So much for a reality check.