Examining Peer-Review

RebornHammer's Campaign

Our good friend RebornHammer dislikes the fact that his detractors call for peer reviewed evidence in support of his claims of Homeopathy. RebornHammer dislikes the fact so much that he has begun to wage a campaign to discredit peer review.

RebornHammer:

tinyurl com/735c5v9

Richard Smith, also a past editor of the British Medical Journal, recently summed up his view of journal peer review as follows:2

“We have little evidence on the effectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.”

RebornHammer - Quote-Mining Richard Smith"

RebornHammer: (in response to the claim that the previous article doesn't discredit peer review or the research that demonstrates Homeopathy doesn't work)

tinyurl com/73x23m3 Where to now for health-related journal peer review?: Excerpt from JLM, June 2011 "despite 25 years of effort, journal peer review processes are themselves not “evidence-based”. This is not to suggest that we should become cynical about peer review, but rather that systems of peer review be recognised as generally not being based upon firm scientific data and that they are as much about politics and ethics as about “evidence”."

RebornHammer - Not Understanding the JLM Article.

Measured Response

Dear RebornHammer, nobody ever said that peer review is the end-all and be-all of scientific evidence.

• Is peer review slow? Yes.
• Are erroneous articles found in journals? Yes.
• Do many laypersons and some scientists uncritically accept the validity of peer reviewed articles that appear in a journal? Yes.
• Do journals reflect the biases and sometimes ideologies of their editors? Yes.
• Could we have something better than peer review? Sure, but we don't have that yet and need to use the best tool available which is peer review (perhaps it might be you, RebornHammer, who discovers/develops a better tool).
• Do flaws in peer review invalidate science? Certainly not.

Why then do we prefer peer reviewed evidence over some random article found on "teh Intrawebs" you may wonder? The main thing peer reviewed journals provide for us is a body of evidence that has a guarantee that most of the articles will be free of basic errors.

Your next question, RebornHammer, should be "How does this body of evidence work?"

As scientists perform their research their results are submitted for peer review. Assuming there are no basic errors in the paper the paper gets accepted for publishing. Sometimes a journal's editorial staff do not practice sufficient oversight and a researcher may have to try to get their paper published in different journals before it finally gets accepted.

Regardless, if the scientists' results are verifiable then the conclusions and results of their paper will be used by other scientists when they explain the results of their own particular papers. Over time the number of papers using these results will constitute a body of evidence that confirms the correctness of the results of the original paper.

On the other hand, if the scientists' results are not verifiable then the conclusions and results of their paper will not be used by many other scientists to explain the results in their own particular papers. This will result in a dearth of evidence for the results of the original paper.

What this means, RebornHammer, is that the articles you quote do not invalidate peer review. These articles warn the people who use peer review, be they scientists or laypeople, to maintain a modicum of skepticism even when citing peer reviewed literature!

Now, you may maintain that I and others accept peer reviewed literature uncritically but tear-apart your non-peer reviewed literature. It is true that we are more likely tear apart your non-peer reviewed literature than any peer reviewed evidence you may cite. It is not true however, that I, at least (though I'm sure the others are the same, but I don't wish to speak their minds for them), don't view the peer reviewed evidence we use as our evidence without skepticism.

Nonetheless, the body of evidence is overwhelmingly against the idea that Homeopathy even works. Too much research has been done on Homeopathy over the past 200 years, too many clinical trials have come back in the 60 odd years since they've been used, to support the notion that Homeopathy works in any way imagined by Homeopaths.

The take home message here is: Despite the flaws and limitations of peer review it is the body of evidence that matters and the body of evidence does not support the Homeopathy hypothesis!

Sourced Articles

The Richard Smith Article

The JLM Article