the nature of evidence or dafuq?

When having online arguments it is often customary to back up your claims with links to supporting evidence. Depending on the level of the argument this evidence may be a newspaper article, a digitized photograph or even a blog posting. For ordinary arguments this works very well and the issue of evidence is quick and easily obtainable.

When the argument involves topics of science then articles from reputable, peer reviewed scientific journals are typically used as evidence. Often, all parties involved in the argument will have access to the same article, or the one with access will read out the relevant portions so that all parties can analyze and judge for themselves what the authors were actually saying, the quality of the author's methodology and whether the author's conclusions were in line with the evidence discovered in the author's research and what the readers themselves actually know.

In addition to this going over the paper itself other papers that verify the findings of a paper's authors are searched for. The more papers in independent research that verify the findings of the originally disputed paper's conclusions the more weight is given to the original paper.

This is how grown up, scientifically minded people argue about science. They go to the source, the research of the scientists themselves.

Arguing with Creationists

Before I dive into this, there are some rational theists, none of whom are Creationists, out there and I wish to acknowledge them and not smear them with the same paintbrush as their more irrational peers.

When arguing with Creationists the nature of evidence often gets called into dispute. Now it's interesting because Creationists want to debate the science of evolution which they often feel is an evil plot, or at best misguided due to human frailties. Science always falls back onto what can be verified and is empirical and often when debating Creationists I insist upon such evidence, preferably from reliable scientific sources. However, it occurs to me that Creationists often do not know what I mean when I say "verfiable", "empirical" or "reliable scientific sources", or they do not wish to know and therein lies the problem.

So, as a service to my Creationist friends, I will provide the following the definitions so that the meanings of these words and phrase are clear to them.

Verifiable

: capable of being verified

Oopsie, it appears that we've run into a once of those problems of using a dictionary. We've been referred to another definition. Alright then.

Verified

: to establish the truth, accuracy, or reality of <verify the claim>

So, when I'm asking for verifiable evidence I'm not asking you to assert that your claim is true. I'm asking you to give me the evidence so that I can establish for myself if the claim is true, accurate or in keeping with what is known about reality. If you're asking me to believe that some sky Djinn "did it" but you can't point the Djinn out to me or show me where scientists have found it in their research then I can't verify it. If I can't verify your claim then your claim is unverifiable and, for lack of a better word, worthless. Sorry.

Empirical

1. : originating in or based on observation or experience
2. : relying on experience or observation alone often without due regard for system and theory <an empirical basis for the theory>
3. : capable of being verified or disproved by observation or experiment <empirical laws>
4. : of or relating to empiricism

If your evidence comes from a book, well that's fine, but books by themselves aren't evidence. I'm sorry, but no matter how many time you tell me your book is correct because your sky Djinn says it is doesn't make your book any truer. Besides, then we're back to having to verify the existence of your sky Djinn and if you can't point it out to me or where scientists have discovered your sky Djinn then your book fails as empirical evidence.

Sadly, the same thing goes for prophetic predictions, or claims of scientific foreknowledge. Unless, for prophecy, you can show me where you holy book makes a prediction that is clear and testable and not actually a prophecy for the people living at that time in that holy book then your claims of prophetic evidence fail. Claims of scientific foreknowledge also have to be clear and testable. I mean, why did the bible, for example, only predict Hitler after he started persecuting Jews and Homosexuals? In the same vein, why did the bible only reveal dinosaurs (it doesn't, but that's another topic) only after scientists started digging up and classifying their fossilized remains?

Reliable Scientific Sources

Okay, I don't have a dictionary definition for this one so I'll have to do my best to carefully explain what I mean by this phrase. When I say "reliable scientific sources" I'm saying several things:

1. Articles from reputable, peer reviewed scientific journals
2. Reputable science magazines

So the first thing you need to do is try to find a supporting article in a reputable, peer reviewed scientific journal...oh dear. By the blank looks on my Creationist friend's faces I seem to have uttered more words that they don't seem to understand. Alright then, let's try some more definitions.

"Reputable

1. : enjoying good repute : held in esteem
2. : employed widely or sanctioned by good writers

If you have some claim that birds aren't related to dinosaurs and you pass me off an article from Answers in Genesis: Answers Research Journal, well that's just not a reputable journal. No scientist that has any value in their field publishes their work in Answers Research Journal. There are a couple of reasons for this 1) if you go to Answers about page the publishers quite clearly state that the bible is the ultimate authority and all scientific research must conform to the bible, 2) Answers rarely publishes any research, mostly just rebuttals of science research that they don't agree with, and the odd actual research they do publish is third rate.

Peer Reviewed

: a process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field

Again, if you hand me Creation Ministries International's Journal of Creation claiming I should read it because it's peer reviewed well I'm not. In a reputable journal peer review is carried out by experts that have been selected to perform the review of a particular paper that has been submitted. The more reputable the paper the more experts they have to draw on. In a journal of low repute, such as Journal of Creation, there are very few experts in any scientific field that are associated with them that they can draw on. Also, as in Answers Research Journal, Journal of Creation states on their about page that the bible is the ultimate authority and thereby limits, if not eliminates, the usefulness of any peer review except to perhaps grammar and spelling.

A good source to look for reputable, peer reviewed scientific journals is PAIS.

Carrying Onward

Now, where was I? Oh yes, I was saying the first thing you should do is try to find if there are any articles in any of the reputable, peer reviewed scientific journal that support your claim. Then read it, not just the abstract, but the whole paper, try to understand it, to absorb it. Try to get inside the heads of the scientists and try to see things the way they saw them. Look at their methodology, did they do things correctly? If you don't understand their methodology, ask for help. Try to get a feel for the equations they're using and the graphs they're displaying. Go over the conclusion, does it fit their premise and does it make sense in light of their research?

Do all of this and you might be able to win points against people who accept evolution, be they theist or atheist. In the process you might learn something new about the world around you too.

The Financing of Drug Trials by Pharmaceutical Companies and Its Consequences

RebornHammer accused me of not reading the meta-study "The Financing of Drug Trials by Pharmaceutical Companies and its Consequences" (Schott et al, 2010) (I referred to the authors as Lieb et al in the discussion). I disagree as I did read it and I think my understanding of the meta-study is better than RebornHammer's.

RebornHammer:

tinyurl com/6qjmgb4 PROZAC is so great!! A study on trials of SSRIs revealed that those with significant results were more likely to be published, sometimes more than once, whereas trials with non- significant results or findings unfavorable to the drug under investigation (intention-to-treat analyses versus per-protocol analyses) were not published

wstevenschneider:

1) This article is not a criticism of peer review or science, it's a criticism of pharmaceutical companies and how they try to thwart the science that exposes inefficacy or side-effects of their products. Lieb et al are encouraging government to take steps to see that this doesn't happen and that science continues to hold pharmaceutical companies accountable for evidence.

2) This in no way is evidence supporting Homeopathy.

RebornHammer:

@wstevenschneider thats evidence for a bias against homeopathy and any research or peer review that might prove it to work..

@wstevenschneider And that is why every time you ask for peer review proving homeopathy you are asking for something that is unlikely to ever exist under the current political/ medical situation on the planet earth.

@wstevenschneider hello? can you read? what this is is a criticism of peer review that is used by the medical community to choose treatments for their patients. The point is that the pharmaceutical industry is out of control, and owns(through contributions and donations) most of the most highly respected peer review for medicine out there. This means that if it's not a pharmaceutical drug it does not get the light of day. Homeopathy is not a pharmaceutical drug.

Let's read "The Financing of Drug Trials by Pharmaceutical Companies and its Consequences" (Schott et al, 2010) together.

Methodology

Fifty-seven independent studies looked at various trials that were funded by pharmaceutical companies. These studies were then organized into a meta-study to get a more comprehensive view of what is going on when pharmaceutical companies fund research.

Results

Incomplete registration
With the aim of facilitating public access to clinical trial data and preventing pharmaceutical companies from influencing the publication of results, in 2004 the International Committee of Medical Journal Editors (ICMJE) made registration a condition for publication in any of the 11 leading medical journals (7): new trials had to be registered by 1 July 2005; those already under way, by 13 September 2005. Meanwhile several reg- isters fulfill the ICMJE standards.

In January 2005 major pharmaceutical organi- zations, among them the Pharmaceutical Research and Manufacturers of America (PhRMA) and the Inter- national Federation of Pharmaceutical Manufacturers and Associations (IFPMA), implemented guidelines that obliged their members to enter trials prospectively in publicly accessible registers (8, 9).

Despite this, two of the 57 studies included in the present investigation contain data suggesting that phar- maceutical companies are still not registering important information on clinical drug trials.
.
.

Evidence that some pharmaceuticals are not disclosing their funding of drug-trials.

Concealment of adverse drug reactions
Seven of the studies investigated concerned themselves with adverse drug reactions (ADRs) in industrially sup- ported drug trials (Table).
.
.

Evidence that some pharmaceuticals are repressing the publication of self-funded trials that reflect negatively on their product.

Publication bias
Of the 57 studies included in this investigation, 14 ana- lyzed the connection between the type of funding of a trial and publication bias.
.
.

Evidence that some pharmaceuticals only present the self-funded trials that reflect positively on their product to the FDA.

Rights over trial data and restricted publication rights
Two investigations on this topic were identified. Both an analysis of the protocols of all studies initiated and published by pharmaceutical companies in a particular region of Denmark in 1994 and 1995 (e28) and a ques- tionnaire survey of medical specialists in Australia (e29) indicated that in some trials pharmaceutical companies secure the rights over the data and place constraints on publication rights.
.
.

Evidence that some pharmaceuticals consider the methodology, results and data from the drug trials on their product to be propriety information, particularly if they are funding the trial.

Ghostwriters and guest authors
A case study on rofecoxib (e5) and an investigation into the above-mentioned studies in Denmark both found evidence of frequent resort to ghostwriters and guest authors in industry-funded publications. The Denmark investigation showed that statisticians employed by pharmaceutical companies are frequently not men- tioned in the published articles (e4).

Evidence that some pharmaceuticals attempt (and succeed in some cases) to control the direction of and what is said in the report of a trail they are funding.

Conclusion

This systematic review clearly shows that clinical trials with the involvement of pharmaceutical companies often present the therapeutic benefit of a drug in too positive a light and also fail to mention risks. Clinical studies are increasingly being funded by pharmaceuti- cal companies (e32­e35). Professional medical bodies construct evidence-based guidelines on the basis of published trial results, so their recommendations may be flawed. This contributes to excessive prescription of expensive new drugs whose efficacy is overestimated and risks underestimated. Moreover, because the evi- dence is distorted patients do not receive adequate in- formation (14).

In the past few years measures have been taken worldwide to deal with the problems described here. Laws have been enacted, for example, with the inten- tion of securing public access to research data (15­18). In the USA, for instance, a law of 27 September 2008 prescribes the registration and publication of the results of clinical trials in a register accessible on the internet (15, 19). In the European Union, directive 2001/20/EC requires registration of all clinical studies (16). A guideline implemented in 2008 lays down what classes of information from the EudraCT database-- accessible only to governmental authorities--should be made available in the publicly accessible EudraPharm drug database, which thus remains incomplete (17, 18).

Pharmaceutical organizations have implemented recommendations that are intended to ensure compre- hensive publication of research findings, whether posi- tive or negative (9, 12, 20). This initiative on the part of the pharmaceutical industry is welcome; however, the present investigation shows that negative results are still not being published in timely fashion and control mechanisms have failed.

Official regulatory measures to guarantee public access to study protocols and results and prevent the withholding of information about dangerous ADRs are urgently required. This would also give independent drug bulletins and bodies representing physicians, e.g., the Drug Commission of the German Medical Associ- ation, the opportunity to obtain detailed, unbiased in- formation about new drugs. Furthermore, it should be obligatory to prove that a new drug provides additional benefit compared with existing pharmacological and non-pharmacological forms of treatment. More public funding should be made available for independent studies (21, 22).

Measures must be taken at many levels to ensure that commercial interests do not undermine the knowledge of scientifically correct study planning, study execu- tion, and publication (4, 5, e15, 15, 23­25, e36, e37). A large number of physicians are involved in the planning and conduct of drug trials. For the benefit of their pa- tients, they should assume greater responsibility and work to counteract the economic self-interest of pharmaceutical companies in research and clinical practice.

This meta-study is not an indication of the corruption of peer review though it does indicate that some pharmaceutical companies attempt to subvert the peer review process in pursuit of profits. What the meta-study does show, albeit indirectly, is the self-correcting nature of peer review at work. Science is always questioning things, even its own conclusions. Peer review is also subject to this scrutiny.

Here we see scientists taking a look at a particular portion of peer review, in this case drug-trials, and finding evidence of pharmaceutical companies trying to subvert the peer review process. This meta-study serves to alert researchers, journal editors, medical doctors and policy makers of these subversion attempts.

Furthermore, the meta-study acknowledges that, while progress has been made countering some of these threats to peer review, more effort is required as more pharmaceutical companies are starting to fund research. Schott et al calls for several things:

• Official regulation of drug-trails to guarantee public access to studies and provide disclosure of adverse drug reactions.
• New drugs should provide benefits beyond that provided by old regimes.
• Increase public funding for more independent studies.
• Improved measures to prevent commercial interests from subverting the research and the peer-review process.
• Doctors to be proactive in countering the commercialization of medical science.

It does not follow that because there are pharmaceutical companies that attempt to subvert research and peer review that the peer review process is unreliable.

In fact, much of the message that comes from this meta-study is that the reader should be wary of pharmaceutical-funded clinical trials. As usual, the trials to go to are the publicly-funded, independent third-party studies that have no vested interest in the success of the drug being reviewed.

Far from being evidence that peer review and medical science are corrupt and unreliable, this meta-study shows us that peer review allows scientists to verify attempts to corrupt the process and to suggest corrections. It also demonstrates that science is out to protect the well-being of the general public and that pharmaceutical companies must continue to be held accountable.

While no one disputes peer review is not without its flaws, it is the only thing preventing pharmaceuticals from corporately raping the public. This meta-study serves as a call-to-arms to protect the integrity of peer review from being subverted by pharmaceuticals whose main interest is producing profit for their share-holders.

So, RebornHammer I'm curious as how you got, "The point is that the pharmaceutical industry is out of control, and owns(through contributions and donations) most of the most highly respected peer review for medicine out there," from Schott et al when this is clearly not what the meta-study says. I get the impression that you read the abstract and drew your own conclusions without bothering to find out what the meta-study actually said.

Examining Peer-Review

RebornHammer's Campaign

Our good friend RebornHammer dislikes the fact that his detractors call for peer reviewed evidence in support of his claims of Homeopathy. RebornHammer dislikes the fact so much that he has begun to wage a campaign to discredit peer review.

RebornHammer:

tinyurl com/735c5v9

Richard Smith, also a past editor of the British Medical Journal, recently summed up his view of journal peer review as follows:2

“We have little evidence on the effectiveness of peer review, but we have considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.”

RebornHammer - Quote-Mining Richard Smith"

RebornHammer: (in response to the claim that the previous article doesn't discredit peer review or the research that demonstrates Homeopathy doesn't work)

tinyurl com/73x23m3 Where to now for health-related journal peer review?: Excerpt from JLM, June 2011 "despite 25 years of effort, journal peer review processes are themselves not “evidence-based”. This is not to suggest that we should become cynical about peer review, but rather that systems of peer review be recognised as generally not being based upon firm scientific data and that they are as much about politics and ethics as about “evidence”."

RebornHammer - Not Understanding the JLM Article.

Measured Response

Dear RebornHammer, nobody ever said that peer review is the end-all and be-all of scientific evidence.

• Is peer review slow? Yes.
• Are erroneous articles found in journals? Yes.
• Do many laypersons and some scientists uncritically accept the validity of peer reviewed articles that appear in a journal? Yes.
• Do journals reflect the biases and sometimes ideologies of their editors? Yes.
• Could we have something better than peer review? Sure, but we don't have that yet and need to use the best tool available which is peer review (perhaps it might be you, RebornHammer, who discovers/develops a better tool).
• Do flaws in peer review invalidate science? Certainly not.

Why then do we prefer peer reviewed evidence over some random article found on "teh Intrawebs" you may wonder? The main thing peer reviewed journals provide for us is a body of evidence that has a guarantee that most of the articles will be free of basic errors.

Your next question, RebornHammer, should be "How does this body of evidence work?"

As scientists perform their research their results are submitted for peer review. Assuming there are no basic errors in the paper the paper gets accepted for publishing. Sometimes a journal's editorial staff do not practice sufficient oversight and a researcher may have to try to get their paper published in different journals before it finally gets accepted.

Regardless, if the scientists' results are verifiable then the conclusions and results of their paper will be used by other scientists when they explain the results of their own particular papers. Over time the number of papers using these results will constitute a body of evidence that confirms the correctness of the results of the original paper.

On the other hand, if the scientists' results are not verifiable then the conclusions and results of their paper will not be used by many other scientists to explain the results in their own particular papers. This will result in a dearth of evidence for the results of the original paper.

What this means, RebornHammer, is that the articles you quote do not invalidate peer review. These articles warn the people who use peer review, be they scientists or laypeople, to maintain a modicum of skepticism even when citing peer reviewed literature!

Now, you may maintain that I and others accept peer reviewed literature uncritically but tear-apart your non-peer reviewed literature. It is true that we are more likely tear apart your non-peer reviewed literature than any peer reviewed evidence you may cite. It is not true however, that I, at least (though I'm sure the others are the same, but I don't wish to speak their minds for them), don't view the peer reviewed evidence we use as our evidence without skepticism.

Nonetheless, the body of evidence is overwhelmingly against the idea that Homeopathy even works. Too much research has been done on Homeopathy over the past 200 years, too many clinical trials have come back in the 60 odd years since they've been used, to support the notion that Homeopathy works in any way imagined by Homeopaths.

The take home message here is: Despite the flaws and limitations of peer review it is the body of evidence that matters and the body of evidence does not support the Homeopathy hypothesis!

Sourced Articles

The Richard Smith Article

The JLM Article