Junk science from farm to fork

This has been doing the rounds on Twitter and deservedly so. It's an excellent demonstration of how quack science is conducted, published and reported.

John Bohannon got sixteen volunteers, a statistician and some chocolate with the hope of finding an association with something - anything - for an epidemiological study.

One group followed a low-carbohydrate diet. Another followed the same low-carb diet plus a daily 1.5 oz. bar of dark chocolate. And the rest, a control group, were instructed to make no changes to their current diet. They weighed themselves each morning for 21 days, and the study finished with a final round of questionnaires and blood tests.

It turned out that the chocolate-eating group lost ten per cent more weight than the other two groups. A newsworthy finding? It shouldn't have been.

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

... With our 18 measurements, we had a 60% chance of getting some“significant” result with p < 0.05. (The measurements weren’t independent, so it could be even higher.) The game was stacked in our favor.

He then submitted it for - cough - peer review...

Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine. It used to be run by the giant publisher BioMedCentral, but recently changed hands. The new publisher’s CEO, Carlos Vasquez, emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”

And awaited the ladies and gentlemen of press...

We landed big fish before we even knew they were biting. Bild rushed their story out—”Those who eat chocolate stay slim!”—without contacting me at all. Soon we were in the Daily Star, the Irish Examiner, Cosmopolitan’s German website, the Times of India, both the German and Indian site of the Huffington Post, and even television news in Texas and an Australian morning talk show.

 That's how it easy it is folks. Do read the whole article.