The return of dodgy minimum pricing modelling

The MESAS evaluation of minimum pricing (MUP) in Scotland has produced a report which uses the IRI sales data. This differs from the Nielsen sales data in various ways. Neither system is perfect (they both have to guesstimate how much alcohol is sold in Aldi and Lidl, for example), but it is useful to see how they compare. 

I had a glimpse of the IRI data in 2019 so I was able to report at the time that off-trade alcohol sales actually rose in Scotland in the first ten months of MUP, although by the end of the first full year there had been a slight decline. 

The IRI figures published this week suggest that very little changed between 2017 and 2019. Remember that MUP was introduced in May 2018 and that the figures for 2020 don't tell us much because the on-trade was closed for much of the year. 

According to the IRI data, there was slightly more alcohol sold in the off-trade in the first full year of MUP (2019) than in the first full year before MUP was introduced (2017). There was also very little change in England and Wales.

The exact figures are as follows: 

Off trade (IRI) England and Wales (litres per adult):

2017: 6.61
2018: 6.71
2019: 6.63
2020: 7.81

Off trade (IRI) Scotland (litres per adult):

2017: 7.67
2018: 7.78
2019: 7.71
2020: 9.04

Off-trade sales in Scotland were 0.04 litres higher in 2019 than in 2017. In England and Wales, they were 0.02 litres higher. By any reasonable standard, there was practically no change. You certainly wouldn't guess that a "world-leading" anti-alcohol intervention had taken place. The Scots bought slightly more off-trade alcohol in 2018 and 2019 than they had in the last full year before MUP was introduced.

The picture is very similar when we look at total alcohol (off-trade plus on-trade). The main difference is that consumption fell in 2020, despite the large rise in off-trade sales.

Here are the figures:

Total alcohol sales (IRI) England and Wales:

2017: 9.29
2018: 9.36
2019: 9.23
2020: 8.88

Total alcohol sales (IRI) Scotland:

2017: 10.51
2018: 10.52
2019: 10.44
2020: 10.01

Here we see less alcohol being bought in Scotland in 2019 than in 2017, but that is entirely down to sales in pubs, restaurants and clubs - which were not affected by MUP - and the same thing happened in England and Wales. Overall, there were 0.06 fewer litres per adult sold in England and Wales in 2019 than in 2017 and 0.06 fewer litres per adult sold in Scotland. Exactly the same.

This shows that you can get different results if you use different data sets. With the Nielsen figures, you can just about see a decline in 2019 that could plausibly be due to MUP. It's not much of a decline and there have been larger ones in some previous years, but it is at least consistent with the idea that MUP reduces consumption.

But with the IRI figures, you don't even get that. Sales barely changed at all between 2017 and 2019, not in the off-trade and not overall either. In fact, alcohol sales in Scotland were lower in the off-trade in 2017 than in any subsequent year - and it is only in the off-trade that MUP has an effect on prices.

And so I was baffled to see the MESAS authors come to this conclusion...

In unadjusted analysis using Nielsen data, the introduction of MUP was associated with a 3.7% (95% confidence interval (CI): 1.8% to 5.5%) reduction in the total volume of pure alcohol sold per adult in Scotland. Using IRI data, a 4.3% (2.1% to 6.5%) reduction was observed.

What?!?

 

In the unadjusted and controlled model that used Nielsen data, MUP was associated with a 6.6% (5.2% to 8.0%) reduction in per-adult off-trade alcohol sales in Scotland while the same model using IRI data resulted in a 4.2% (2.9% to 5.6%) reduction.

Eh?!? What were the inputs of this model?

 

In the fully adjusted and controlled model using Nielsen data, a 6.6% (5.1% to 8.0%) reduction was observed compared to a 4.0% (2.7% to 5.3%) reduction when using IRI data.

None of this seems to make sense. Clearly there was not actually a 4% reduction in sales. The model seems to be comparing sales in Scotland with sales in England and Wales. That's fair enough. However, there would have to be a big rise in England and Wales for a model to suggest even a relative decline in Scotland - and there wasn't.

The MESAS authors have access to more data than they have published. The key to understanding their claim is that it refers only to the first year of MUP, i.e. May 2018 to April 2019. Their model ignores what happened afterwards and they only provide figures for calendar years.

It would be interesting to see the full data and it is odd that they haven't published it. It is possible that there was more of a spike in sales in England and Wales between May 2018 to April 2019 than there was in Scotland. All we can say for sure is that if there was any effect from MUP, it was very short-lived because more off-trade alcohol was being sold in Scotland in 2019 than in 2017.  

I'm not saying the IRI figures are necessarily more accurate than the Nielsen figures. I really don't know. But I challenge anyone to look at the graphs and figures above and tell me they can see an effect from minimum pricing.  

It was dodgy modelling that got Scotland into minimum pricing in the first place and it looks like dodgy modelling is going to be used to justify it in retrospect.