Thursday, 28 October 2010

Argumentum ad peer review

A good article by Sandy Starr about the argumentum ad peer review in Times Higher Education, which echoes what Brendan O'Neil wrote in Spiked back in August.

This part particularly appealed to me, for obvious reasons:

Once politicised in this way, peer review becomes a sort of quarantine, where ideas can be contested only by a select few before being presented to us as a fait accompli. Take, for example, the attitude of epidemiologists Kate Pickett and Richard Wilkinson, authors of the book The Spirit Level: Why More Equal Societies Almost Always Do Better. They state imperiously: “Almost all of the research presented and synthesised in The Spirit Level had previously been peer-reviewed…In order to distinguish between well-founded criticism and unsubstantiated claims made for political purposes, all future debate should take place in peer-reviewed publications.”

Such disdain for arguments motivated by “political purposes” speaks (non-peer-reviewed) volumes.

As I have said before, there is no better alternative to peer review and there is nothing inherently wrong with it. The only problem with peer review is its public perception as a stamp of absolute truth. While honest academics in any field tend to be realistic about their limitations, something about human nature requires things to be seen in black and white, good and evil, right and wrong.

That peer review is merely a form of editorial control, which—at times—represents nothing more than allowing colleagues to publish an article the house magazine is a truth that is lost in the rush for certainty. As a rule of thumb, anyone who defends their work on the appeal to peer review and the appeal to authority alone, while shying away from debate about the work itself, should be treated with suspicion.

Starr concludes:

Peer review is valuable and worth defending, but only inasmuch as it promotes impartiality and standards within specialist fields. It should not be used as an arbiter of how we run our affairs. We already have a system for that – it’s called politics. And we are all qualified to participate in it, by virtue of being born human.

Please go read.

Sandy Starr will be appearing at the Battle of Ideas in London at the end of the month. Should be interesting.


PJA said...

Yes. Wilkinson and Pickett have some gall in writing a mass market book and then going 'ner ner you're not qualified to argue you ignorant plebs, we'll only talk about it with people who drink our Kool-Aid'.

But what do I know, I'll only debate my work in the pages of Viz.

Carl V Phillips said...

In the work I am doing on the health effects of wind turbines (see my blog if you are interested), the "experts" on the other side are fond of denying the evidence of problems because "it is not peer reviewed!". (Some of it actually is, some of it is not, but none of it would be any different if it were.) But at the same time they put all their faith in promulgated guidelines for what is "safe" that are based on pure speculation. Naturally, those are not generally peer reviewed.

Otoh, sometimes utter speculation like that is actually peer reviewed. Anyone with friends at a journal can put together a purely speculative "consensus report" (i.e., what me and my friends say), based on no scientific analysis, and turn it into a "peer reviewed publication".

(Sorry -- kind of a random rant. It happens to be what I have been working on for the last week.)

Anonymous said...

Peer review in it's current incarnation does not "promote impartiality and standards within specialist fields."

Anonymous said...

Perhaps, I'm stating the obvious, but I find this though re-occurring to me lately.

Like Gertrude Stein wrote: "A rose is a rose is a rose."

Science (to grossly oversimplify) is the study of phenomena; it is not absolute truth about the phenomena in question.

For instance, the economy is not what economists say about it; the economy is the economy is the economy. The same for history, climate, diet, etc.

The rush to certainty you describe will inevitably come up against the hard fact that, at some point or another, you can't force any phenomena or interaction to be what you want it to be.

For instance, if you ask someone "Do you believe in Global Warming?" you are really asking a dozen a more involved questions. Somehow though, it seems that the very same people who constantly invoke "science" are the very same people who demand that the question be answered with an unthinking and unequivocal "YES!" These same people don't regard questioning as healthy inquiry, but as an annoyance. Skepticism is shoved aside under cheap labels like "denialism". Meanwhile, the climate doesn't care what anyone thinks about it.

At some point, our feelings and debates on these matters will be negated by often discussed but rarely acknowledged reality. It would be nice if science stopped retreating to the fallback position of the precautionary principle, with all of its political implications, and got busy again caring about what is being studied.