Motivated reasoning - a case study

Last week, the IEA published my report Fast Food Outlets: What is the Evidence? As the title suggests, it is a review of the published evidence on the proximity and density of fast food outlets to obesity. It shows that only 20 per cent of the 74 studies conducted have found a positive association between fast food availability and body weight. Of the studies related to children, only 15 per cent have found a positive association.

This is no secret to those who are familiar with the evidence. Several previous evidence reviews have come to much the same conclusion.

Williams et al. (2014: 372) ‘did not find strong evidence at this time to justify policies related to regulating the food environments around schools’. Gordon-Larsen (2014) found that: ‘Studies of access to fast food and body weight generally showed null results’. Cobb et al. (2015) found that two-thirds of the associations between fast food availability and obesity in the literature were null, as did Mackenbach et al. (2014: 12) who took the methodological quality of the studies into account when conducting their review before concluding that ‘the overall evidence for an association between environmental factors and weight status is weak.’

Public Health England and its minions in local authorities have chosen to either ignore the academic literature or cherry-pick the few studies that appear to support 'zoning laws' (ie. banning new fast food outlets opening in certain areas). If they are not going to be honest about what the evidence says, it is up to people like me to shine a light on it.

This hasn't gone down well with a man called Greg Fell. Greg is Sheffield's Director of Public Health and is on £106,000 plus expenses and benefits (his predecessor was on an almost unbelievable £178,000). He is not some hikikomarxist living in his mum's basement. It is easy to forget that as you read his blog post.

Given the source, I’d guess it’s been funded by KFC, maybe the Colonel himself.

This is Fell's opening gambit. Going straight in with the ad hominem is what the bottom feeders on Twitter do and that is about Fell's level. It's his 'guess' that the IEA report was 'funded by KFC'. I can tell you categorically that it wasn't. The IEA doesn't do commissioned research and I've never heard of KFC, or any other fast food outlet, giving money to the IEA in all my years working for them. I can tell you with 100 per cent certainty that the first time anybody in the fast food industry, from the biggest burger chain to the smallest kebab shop, heard about this report was the day it was published.

But let's pretend for a moment that the IEA is mostly funded by McDonald's and that this particular report was commissioned by Pizza Hut. Hell, let's say it was written by Pizza Hut. Would its findings be any more or less true? Of course not. Leaving aside the fact that the report is only a summary of studies written by other people (mostly 'public health' people), the validity of a study's conclusions cannot be judged by the identity or motivations of its author. If it turns out to be a pack of lies then the motivations of the author might explain why, but these motivations do not automatically make it a pack of lies. This is basic critical thinking.

However, as I say, no such commercial interests were involved in this report so Fell's 'guess' is worthless. But that doesn't stop him putting it at the heart of his blog post's title:

The McKentucky-Hut review of fast food zoning evidence

It is worth noting how casually these people lie. Whether something is true or false seems wholly irrelevant to them. The only thing that matters is the effect that saying it will have on the reader.

Having indicated that evidence is not really his thing, Fell confirms it...

Bluntly, I can’t be bothered to go through it line my line. It seems of pretty poor quality academically speaking – a bit opaque re methods, selective quoting, flawed lines of argument. Of course maybe it doesn’t present to be an academic work, but it will still be presented as evidence to decision makers – who may or may not be able to pick through the flaws.

Not only is Fell not prepared to 'go through the it line by line' (ie. read it properly), he is apparently too busy to explain what the 'flaws' are. What are the 'flawed lines of argument'? What examples can he give of 'selective quoting'? Alas, he can't be bothered to provide a single example.

Or, more likely, he can't find any 'flaws' in a report that is mostly just a list of studies and a summary of their conclusions, but he has seen critiques of other people's work that use these phrases and thinks that he can poison the well by repeating him here. 

Some extracts of the studies reviewed are given. Obviously there’s no way of knowing whether the extract actually represents the conclusion of the original study.

I laughed out loud when I read this. Obviously there is a way of knowing whether the extract reflects the conclusion. You can read the study. To Fell, the idea of reading an academic study all the way through is not just alien, but completely unthinkable.

I don't expect readers to go through 74 studies to check that my conclusion is sound. I know better than anyone what a long and tedious process it is. On the other hand, if you're going to effectively accuse me of lying then the onus is on you to read the evidence and prove it.

What you don't do is say that you can't bothered to read the evidence while showing that you can be bothered to write a blog post implying that I'm making it all up. That would be true of any blogger, but it is doubly true of a Director of Public Health who should be familiar with the evidence anyway.

But who needs evidence anyway?

It is important we don’t become our own worst enemy in the pursuit of evidence.

As I said in a prior blog, Lack of evidence is frequently cited as barrier for not doing something. This is fine, but that must be considered in terms of counterfactual – what’s the evidence for the counterfactual or maintaining the status quo. Is doing noting an option. Establishing a level of burden of evidence proof in a complex system is different. Some say we need to move to a decision logic framework, not hypothesis testing – ie not a criminal burden of proof, but balance of probability and what is happening in the background.

But there is not a 'lack of evidence' in this area. There are 74 studies looking at this specific issue. There may be a lack of evidence to support the policies for which Fell advocates, but that is not remotely the same thing as a lack of evidence. If you have twenty well-conducted, peer-reviewed studies looking at mobile phone use, for example, and none of them found an association with autism, it would be dishonest to claim that there is a lack of evidence or that 'we just don't know'. We would conclude that mobile phone use does not cause autism. If one or two of them found a weak association, we would conclude that, on the balance of probabilities, mobile phone use does not cause autism.

It would obviously be wrong to claim that people who live near fast food outlets were more likely to be obese if no research had ever been conducted. But it is arguably even more wrong to make this claim after dozens of studies had been carried out all over the world for fifteen years and the majority of them had failed to find any such association.

There will always be those who are opposed to a policy proposition, sometimes on the basis of evidence, sometimes on pure ideological or commercial grounds.

Whether the Planning Authority should pay any attention to it is also debatable

The Authority might ask who sponsored the IEA to do this review, and why. 

Accusing someone of being 'ideological' is what Twitter bottom feeders do when ad hominems fall on stony ground. If the IEA presents evidence that free markets are the solution to a certain problem, an imbecile might respond by pointing out that the IEA is a free market think tank and therefore 'would say that, wouldn't they?' The appeal of this line of attack is the same as that of the ad hominem. It allows idiots to dismiss evidence without having to deal with its substance, or even read it.

In the lines quoted above, Fell commits both fallacies. He thereby concludes his blog post in the way he started it, with not a single substantive criticism of the research and with a claim about funding which is simply untrue.

I will leave the reader to judge who is being 'ideological' here but, as my colleague Kristian says, Fell's response is a wonderful example of a dull mind in motivated reasoning mode.