“The Illusion of Evidence-Based Medicine” investigates corruption in many areas of so-called evidence-based medicine. I found the author’s work examining Randomized Controlled Trials to be the most telling and damning. But there’s more of which I’ll touch on.
Clearly, medical treatments should only be administered if they have been proven to be safe and effective.
That just makes sense.
It is an axiom of medical research that the best evidence of safety and effectiveness can be found through a large, controlled, randomized, double blinded trial.
Alas, there is a problem.
Many experts are reaching the same conclusion; large, randomized clinical trials are very expensive and often only affordable by the pharmaceutical industry.
So, the question must be asked: does the profit motive of Big Pharma undermine the reliability of RCTs?
Some of the top people in the medical profession believe that is the case. Stanford professor of medicine, John Ioannidis, had this comment, “The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.”
The former editor of the prestigious New England Journal of Medicine agreed. “It is simply no longer possible to believe much of the clinical research that is published."
Is medical science being compromised? And if so, how?
This question is the strength of Jon Jureidini and Leemon McHenry’s book.
The Actual Process of RCTs
First, let's talk about how RCTs are conducted.An “investigator” - someone or some group who wishes to conduct a study - will contact a sponsor to fund the study. In drug trials, the sponsor is often a pharmaceutical company.
The investigator will propose a “study design” to the sponsor.
The design may be a RCT comparing the safety and efficacy of one drug against another drug or against a placebo. And it will have what are called “primary endpoints” and “secondary endpoints.”
The primary endpoints represent the hypotheses being tested. For instance, if a drug is designed to prevent allergy-related deaths then the primary endpoint might be: Did the drug succeed in preventing deaths when compared to the control group?
Questions like that are typical primary endpoints.
A secondary endpoint is a finding from the study which is collateral to the purpose of the study but may be of interest. For instance, a drug designed to prevent allergy-related deaths might also have a secondary endpoint such as “Was quality of life improved?”
But remember this. The study was designed to answer a binary question (death or survival.) The study was not designed to resolve the ambiguous, non-binary secondary endpoint like quality of life. Secondary endpoints may raise interesting issues, but they usually don’t prove anything.
This is important, we’ll come back to it later.
The actual conduct of the study is guided by rules called “protocols.” Protocols include:
- Who qualifies to participate (selection criteria);
- How many people will be part of the study;
- How long the study will last;
- How the drug will be given to patients and at what dosage;
- What assessments will be conducted, when, and what data will be collected;
- How the data will be reviewed and analyzed.
What could go wrong?
The answer to that question is: Just about everything.
Let’s look at each step of the RCT according to the authors.
Study Design
A study can be deliberately designed to succeed or fail - depending on the desired outcome.For instance, if a drug is meant to look good in the trial, it can be tested against a drug known to be ineffective. Or it can be tested against another drug administered with a dose too low to have any effect.
Either way the drug meant to be promoted looks good by comparison.
If, on the other hand, if a drug is meant to look bad, the dosage of that drug can be set so high as to increase adverse events. On the other hand, the dosage can just as easily be set too low so it would not have any real effect.
Either way, the drug can be demonized.
Other scenarios are easy to imagine. For instance, the time of administration of the drug is important.
Some drugs must be administered early in a disease process to be effective. Typically, flu medications must be administered early.
There are other drugs where the time of administration is important. Thus, the time of the administration could be manipulated in a study in order to show purported effectiveness or ineffectiveness depending upon the desired goal.
Medical professionals argue that there are other ways by which studies have been designed to come to a pre-determined conclusion. They submit that it can be done by locating a study in a locale where the availability of commonly consumed over-the-counter drugs will engineer the desired result. Or by the selective utilization of the criteria for inclusion or exclusion of participants.
In short, it is easy to imagine any number of ways that a study design can generate a preferred outcome.
Protocols, Data Collection and the Final Report
After the design of the study is approved, next comes the administration of the drug and the collection of the data. Once that data is collected it is collated into a final clinical study report.That report can be made to actually hide the real results from the study. This can be done via the study’s rules or protocols. For instance, adverse events can be white-washed out of a final clinical study report by a categorization protocol. Here is what that means:
The final clinical study report simply does not list all the adverse events shown by the raw data. Instead, the protocols of the study may require that adverse events be reported by way of pre-defined categories.
The book offers an example of how adverse events in a the study of an anti-depressant were hidden by “categorization”. Under the protocols of that study there was no category for suicidal thoughts. Thus, suicidal thoughts were, of necessity, reported in another category, “emotional lability." This, in effect, disguised severe adverse events (suicidal thoughts) by reporting it in a seemingly benign category of emotional lability.
Another way to hide adverse events is by setting arbitrary reporting thresholds. For instance, the protocols may specify that adverse events will only be reported if they are found in more than 10% of patients. In that way, even very serious adverse events might not be reported at all since not enough of them occurred to meet the reporting threshold.
It gets worse.
Reporting thresholds can be coupled with categorization to further limit the reporting of adverse events. Suppose a study has no category for suicidal thoughts. In that case, suicidal thoughts might sometimes be reported under emotional lability and sometimes under “agitation.” By dispersing suicidal thoughts under two categories, neither category might reach the 10% threshold for reporting and, accordingly, the problem would not be reported at all.
This strategy can hide even dramatic signals regarding adverse events. Suppose there was a 19% rate of suicidal thoughts. That is a substantial signal indicating significant danger in the use of the drug, but if those reports of suicidal thoughts are spread equally between two separate categories, e.g. emotional lability and agitation (9.5% each) neither category ever reaches the 10% reporting threshold. Thus, the fact that 19% of test subjects reported suicidal thoughts is not reported at all. That effectively hides a very significant problem which, in reality, far exceeded the 10% threshold.
Non-reporting can also be facilitated if a clinician unilaterally fails to report the adverse event at all. The clinician may simply decide that an adverse event is not related to the drug being tested and therefore does not have to be placed in any category whatsoever.
In these ways, adverse events can simply be whitewashed out of a study altogether.
What About Honesty?
Now, at this point you might say, "well, wait a minute, some of these authors of the articles generated by the study must be honest and certainly some of those honest authors will object to the deliberate mischaracterization or hiding of relevant data."That is a legitimate objection. But prepare to be shocked.
Oftentimes, the authors of these articles are not even aware of the actual data at all.
Yes, you read that correctly.
How can that happen? Like this:The study may have been conducted by a contract research organization and the article itself written up by professional medical ghostwriters.
The actual authors may be barely involved in the study at all.
One medical ghostwriting company advertisement described the process as follows: The first step is to choose the target journal best suited to the manuscript’s content, thus avoiding the possibility of manuscript rejection. We will then analyze the data and write the manuscript, recruit a suitable well-recognized expert to lend his/ her name as author of the document, and secure his/her approval of its content.That really happens.
But, you might object “well, so what, the authors can see the actual raw data. And some wouldn’t sign on to something that misrepresents that data. Right?”
Not so.
First, remember that the original raw data is compiled into a final clinical study report. And remember that that final clinical study report is generated after the protocols have sanitized the raw data. In reviewing the clinical study report the authors are not seeing the raw data at all.
And now for the clincher…
The authors only get to see the final clinical study report. They don’t get to see the raw data at all. According to Jureidini and McHenry, the raw data itself is considered to be “owned” by the pharmaceutical companies. It can only be released to the purported authors with the consent of the pharmaceutical company.
What? Would researchers actually sign off on a study without seeing the data?
Yes. It happens.
In the Surgisphere scandal eminent scientists unknowingly signed off on a study based upon clinical findings that apparently never existed in the first place.
In short, the authors of a paper might not know if the study had been handled correctly. They might not know if the conclusions of the study are consistent with the actual data. They might not even know if the study was conducted at all. (NOTE: the doctors involved in the Surgisphere article withdrew their endorsement of the study once it become known that there were problems.)
But, yes, it gets even worse.
Jureidini and McHenry relate the story of a researcher who did indeed have access to the original data. She found out that the drug being tested was unsafe and ineffective. Being honest and dedicated she complained. So, the Pharma company simply withdrew funding and terminated the study. But the researcher, Nancy Olivieri, decided to go public and publish her findings. Her actions came with a cost. After Olivieri published, she was sued by the drug manufacturer for an alleged violation of a confidentiality agreement she had signed. It took many years for the legal issues to resolve.
So, here is Jureidini and McHenry’s take on published RCTs.
Even though an article may list a dozen or more authors, the “authors’” involvement in the study may be minimal. They may have read the article, offered some minor editing, but that is it. They may have no idea if the study is actually supported by the raw data. They may have no idea if the study was even conducted.
But, on the other hand, if a researcher actually obtains information about the real data (such as Nancy Olivieri) they may find themselves mired in a serious legal and ethical dilemma. And should they choose honesty and ethics, it can involve them in a multi-year lawsuit.
And, yet, there is more.
Suppose that the results shown by the actual raw data are so horribly awful that that even a dedicated ghostwriter cannot possibly spin it in a way that saves the day.
That happens. The following email from a ghostwriter referred to a manuscript commissioned in relation to a treatment for panic disorder. The ghostwriter noted, “There are some data that no amount of spin will fix …”
Stop!
Fixing data with “spin?” Isn’t this supposed to be science? Well, maybe not.
What Problems?
In any event, problems like horrible data can be still be circumnavigated by Big Pharma.Here is how this is done, according to Jureidini and McHenry.
Suppose that the primary endpoints show the drug is not safe. Further suppose the primary endpoints show that the drug is not effective. What is done then?
Sometimes the response is to simply to drop the primary endpoints from the final article altogether.
But if that is done, then what is left to be published in the medical journal?
Well, at that point, carefully selected positive secondary endpoints are touted as proving efficacy and/or safety. And that happens even though secondary endpoints are not necessarily dispositive of anything in the first place.
Whew!
So, how many RCT’s suffer from these flaws? No one knows for sure. “The prevalence of ghostwriting for the medical journals is unknown precisely because ghostwriting is designed to be untraceable,” wrote Jureidini and McHenry.
And that is just the lowdown on RCTs. There is much more in this book.
And There's More
For instance, pharmaceutical companies have thinly disguised marketing plans for drugs which include presentations at (and sponsoring of) medical conferences, participation in (and sponsoring of) continuing medical education, as well as sponsoring of consumer organizations likely to favor the prescription of a drug.yes, this gets worse, too.
Even formal, published drug recommendation guidelines are often influenced by pharmaceutical companies. The book notes that authors of clinical practice guidelines have extensive conflicts of interest with the pharmaceutical industry. In one study, of the 192 authors of these guidelines surveyed, 87% had interactions with industry, 58% received financial support and 38% had been employees or consultants of industry, report Jureidini and McHenry.
And pharmaceutical companies also influence government regulatory authorities, universities and medical journals.
By way of example, medical journals receive substantial advertising and other revenues from pharmaceutical companies.
The authors state that many journals could not even survive without that income. The book notes multiple instances where medical journals were confronted with “undeniable evidence of fraud,” but the journals nonetheless refused to retract industry studies.
But all the above is just an overview. It is just an outline. There is a lot more in the book including failed attempts at reform, suggestions for effective reform and a philosophical discussion of the difference between actual science, marketing and pseudo-science.
So, take a look.
And the next time someone tells you to “just follow the science,” loan them a copy of this book. Then tell them to read this book “because while I would love to ‘follow the science’, you know, somehow I just can’t seem to find it.”
This is a companion discussion topic for the original entry at https://peakprosperity.com/a-real-problem-the-illusion-of-evidence-based-medicine/
