In The Independent, Nigel Hawkes takes on the argumentum ad peer-review. While I think it's a bit much to say that "peer-reviewed journals aren't worth the paper their printed on", it is certainly true that peer-review does not bestow a stamp of truth, nor was it ever intended to be. And, as regular readers will know, there are plenty of individual studies that aren't worth the paper they're printed on.
The truth is that peer review is largely hokum. What happens if a peer-reviewed journal rejects a paper? It gets sent to another peer-reviewed journal a bit further down the pecking order, which is happy to publish it. Peer review seldom detects fraud, or even mistakes. It is biased against women and against less famous institutions. Its benefits are statistically insignificant and its risks – academic log-rolling, suppression of unfashionable ideas, and the irresistible opportunity to put a spoke in a rival's wheel – are seldom examined.
This didn't matter much when peer review was just a private game for academics. If it made them feel better, that was nice. But in a sinister development, people who have published provocative or implausible claims in peer- reviewed journals have started arguing that they won't listen to criticism unless it has undergone the same laying-on of hands.
The most notorious example of this comes from the authors of The Spirit Level, a book that argues that everything from health to the level of crime, and from happiness to the quality of the toast, is better in societies that are more equal.
Professors Kate Pickett and Richard Wilkinson may well be right in this claim, though I dare say that we can all think of a few exceptions.
What isn't right, however, is their claim that as "almost all" the research that went into The Spirit Level was peer-reviewed, all future debate on the subject should take place in peer-reviewed publications. Come off it, profs, you're pulling our leg, surely? If you're writing a book about a hugely political subject such as inequality, you've surrendered any right to hide behind the flak-jacket of peer review.
I wrote in the comments:
As one of the aforementioned critics of The Spirit Level, the 'argumentum ad peer-review' falls flat for two reasons:
Firstly, the claim that "almost all" of the research in the book has been peer-reviewed is deeply misleading. It is not the research, but the conclusions, that count. Most of the studies cited do not even mention inequality. Instead, they say x has gotten worse, x is associated with y, and then Wilkinson and Pickett jump in to say that y is caused by inequality. That crucial third assumption is where it breaks down. Making conclusions based on other people's work is not the same thing as having those people on your side. There is a much larger body of peer-reviewed research telling us what really causes x and y, but it goes unmentioned in The Spirit Level.
Secondly, it is foolish to think you can take your message directly to the public with a populist paperback and a media blitz, and then demand the public reply only via peer-reviewed journals. Either you stay in the academic ivory tower or you engage with the people. You can't have it both ways.
Hawkes is the director of Straight Statistics (straightstatistics.org), which is a new site to me but looks worth keeping an eye on. The most recent article questions to claims made about the effect of banning 16 and 17 year olds from buying cigarettes. See what you think.
And finally, blogger Anna Racoon is leading a campaign to help a 70 year old widow who has been told to pay £2,500 (plus expenses) for dropping what can only be described as a tiny bit of cigarette ash in the street. Words fail me.
As a, now retired, peer-reviewed and published scientist, I'm fond of telling people who consider peer-review to be the "gold standard" of scientific probity that, in essence, peer-review is very little different from allowing GCSE pupils to mark their friends' exam papers. :-)
ReplyDelete