The Cardinal Sins of Skewed Research, Part 3: Sweeping

If scientific research is to be useful, it needs to be honest and transparent. When a researcher formulates a hypothesis to test, defines the outcome measures, devises the protocol, and carefully collects the data, he or she is duty-bound to the cause of good science to publish the results — or to at least try to do so, since in the peer review system of journal publication that currently exists, there are gatekeepers you must first navigate in order to do it. Whether the outcome refutes or confirms the original hypothesis, we all need to know it; it shouldn’t be swept under the rug. A negative or unexpected result is as important as a positive or anticipated one for the collective advancement of science toward a point that more nearly approaches truth.

But that isn’t always the way of published research.

For example, the proportion of published literature showing positive results increased from about 70% positive (i.e., hypothesis-confirming) outcomes in 1990/1991 to almost 86% positive in 2007 (1). And the increases occurred across all disciplines and countries. Surely this vast proportion of positive results isn’t merely a consequence of prescient researchers making excellent hypotheses that bore corroborative fruit 85% of the time. At one level, it probably reflects the tendency of scientific journal editors to favor the publication of positive results, which get more citations than negative outcomes. But there can be other reasons negative or unanticipated results don’t make the “papers,” some perhaps legitimate and others possibly more nefarious. Among them, per Ana Mlinarić and her team at Rijeka University School of Medicine:

• The study is too small and lacks power. Findings inconclusively suggest no effect.
• Despite large enough sample and well-planned study, findings clearly suggest no effect.
• Instead of desired outcome study produces the opposite effect entirely. (1)

The remedy in all three cases is to publish. Let us see the good, the bad, the ugly, and the negative. Publish the results of the small study (even if it isn’t in a peer-reviewed journal) and maybe other scientists will try the experiment again with perhaps more power. A few such sites do exist (BMC Psychology, PLOS ONE, Journal of Negative Results in Biomedicine, and the fledgling All Results journal, for instance) that welcome null, inconclusive, or even negative results, giving an opportunity for all completed research to be published for everyone to see (2).

In the second case, it is absolutely imperative to publish a good, well-planned study that failed to find an effect — e.g., refuted the original hypothesis. That’s critical information we need to have, and not publishing it introduces “positive publication bias” into the body of research out there and means that the pool of data other scientists and interested parties can draw from is hopelessly skewed. Though there were an estimated 2.5 million articles published in 2014, for instance, that number represents a fraction of the work that was done and not published. That means there is a hole in the fabric of our knowledge.

The last case of non-reporting is the most problematic, because it is frequently the consequence of financial conflicts of interest and self-serving bias. The data didn’t produce the desired outcome, so it just disappears.

Perhaps the most infamous example of sweeping unfavorable data under the rug can be found in the decades-long behavior of Big Tobacco in suppressing mounting industry research that it knew implicated cigarette smoking as a cause of lung cancer. (Obviously this example is does not represent a failure to publish a non-finding or negative result, but it is a case of sweeping important, causal, business-damaging findings under the rug, and that is arguably far worse.)

According to a 1995 report by UC Davis, when the data pertaining to cigarette use and lung cancer risk became too overwhelming to suppress, Big Tobacco turned to suppressing data about the dangers of second-hand smoke in the same manner (3). Here is a case of having strong positive results verifying the hypothesis — Does just breathing cigarette smoke cause lung cancer? — and sweeping it out of sight. This shameful practice sadly is far from unique to Big Tobacco.

Big Pharma learned from its tobacco cousins and has applied the broom with vigor over the decades.

Take, for example, the pharmaceutical giant, AstraZeneca, which in 2010 paid a whopping £125 million settlement to 17,500 schizophrenic patients who alleged the drug company hid damning information about its drug Seroquel causing diabetes (4). The company settled before trial, potentially to avoid shining the bright light of litigation discovery on all its internal emails, correspondence, and other documents. It was the process of legal discovery, after all, that opened Pandora’s box of what Big Tobacco knew and when it knew it. Who knows what lurks, unpublished, in those internal files?

Almost as glaring (and in some ways perhaps as harmful) as the tobacco industry hiding data damning to its claim that cigarettes were safe has been the decades-long pretense that the Framingham Heart Study proved the connection of dietary cholesterol, serum cholesterol, and heart disease when the actual results were quite the opposite. As you know from the Framingham Heart Study series on CrossFit.com (parts 1, 2, and 3), the unexpected results that were exculpatory to dietary cholesterol and fat as causes for cardiovascular disease never were published officially. Were it not for one of the study’s statisticians, who had enough integrity to question the non-publication, we’d never have had access even to what we have of the original study, which is a privately printed monograph that barely saw the light of day. After all the hoopla that attended the carrying out of Framingham, to have attempted to lift the corner of the rug and sweep the negative findings out of sight was (and is) unconscionable.

In 2008, pursuant to an FDA mandate, the United States National Library of Medicine and National Institutes of Health launched a registry database called ClinicalTrials.gov in an attempt to combat this practice of sweeping. Researchers executing both federally and privately funded clinical trials on medications are required by the mandate to enter their study information, eligibility criteria, outcome measures, and protocols before beginning to study the first patient. They then are required to submit their results within a year of trial completion. The aim is noble, but the execution is far from perfect.

According to Nature, although research on drugs approved by the FDA is required to be submitted to ClinicalTrials.gov within a year of completion or researchers risk losing grants and even being fined as much at $10,000 per day, half these studies never make it into print in medical/scientific journals, the venue through which most doctors and researchers learn about the latest research. So, the drug companies maintain compliance with the FDA but still don’t publish their results to the research world at large via journals.

Even those trials reported in journals often were reported inaccurately, especially in terms of negative side effects of the drug under study. In one example, in a survey of 600 studies randomly selected from the ClinicalTrials.gov database, only 50 percent actually were published. Of those published, negative side effects were described only 45 percent of the time in the journals but 73 percent of the time in the database. So, there is more fudging of results in the journals, which most of the medical and scientific community read, than in the database entries that are required by law.

Non-publication has long been a recognized issue confounding the validity of the published body of research. As early as 2005, the International Committee of Medical Journal Editors (ICMJE) made an attempt to curb the practice. The committee vowed to publish only the results of trials that were fully registered before commencing and fully compliant in reporting results. But in the period from 2009 to 2012, still as many as half the published trials were not properly registered and reported.

As late as 2015, according to a STAT survey, registrations still were lagging. Stanford University, for example, had only registered and reported results on about 30% of its trials that year. After being called out on it, the university submitted results on almost 85% by 2017 (5). Memorial Sloan Kettering Cancer Center went from reporting a mere 20% of its results in 2015 to 98% by 2017. So things do seem to be improving on this score.

It’s a flawed system, to state the obvious. And it still could use a liberal dose of disinfecting sunlight. They say a new broom sweeps clean; perhaps we need a new one that sweeps research back out from under the rug.

Additional Reading

• The Cardinal Sins of Skewed Research, Part 1: Cloaking
• The Cardinal Sins of Skewed Research, Part 2: Racking
• The Cardinal Sins of Skewed Research, Part 4: Bait and Switch
• The Cardinal Sins of Skewed Research, Part 5: Burning Britches

Drs. Michael and Mary Dan Eades are the authors of 14 books in the fields of health, nutrition, and exercise, including the bestseller Protein Power.

Dr. Michael Eades was born in Springfield, Missouri, and educated in Missouri, Michigan, and California. He received his undergraduate degree in engineering from California State Polytechnic University and his medical degree from the University of Arkansas. After completing his medical and post-graduate training, he and his wife, Mary Dan, founded Medi-Stat Medical Clinics, a chain of ambulatory out-patient family care clinics in central Arkansas. Since 1986, Dr. Michael Eades has been in the full-time practice of bariatric, nutritional, and metabolic medicine.

Dr. Mary Dan Eades was born in Hot Springs, Arkansas, and received her undergraduate degree in biology and chemistry from the University of Arkansas, graduating magna cum laude. After completing her medical degree at the University of Arkansas, she and her husband have been in private practice devoting their clinical time exclusively to bariatric and nutritional medicine, gaining first-hand experience treating over 6,000 people suffering from high blood pressure, diabetes, elevated cholesterol and triglycerides, and obesity with their nutritional regimen.

Together, the Eades give numerous lectures to the general public and various lay organizations on their methods of treatment. They have both been guest nutritional experts on over 150 radio and television shows, including national segments for FOX and CBS.

References

1. Mlinarić A, Horvat M, and Smolčić VS. Dealing with the positive publication bias: Why you should really publish your negative results. Biochem. Med. 27.3(2017).
2. Mudrak B. Negative results: The dark matter of research. AJE Scholar. https://www.aje.com/arc/negative-results-dark-matter-research/. Accessed 8 April 2019.
3. Big tobacco and science: Uncovering the Truth. UC Davis Health. https://health.ucdavis.edu/welcome/features/20071114_cardio-tobacco/. Accessed 8 April 2019.
4. Goldacre B. Drug firms hiding negative research are unfit to experiment on people. The Guardian. https://www.theguardian.com/commentisfree/2010/aug/14/drug-companies-bury-negative-research. Accessed 8 April 2019.
5. Piller C, and Bronshtein T. Faced with public pressure, research institutions step up reporting of clinical trial results. STAT. https://www.statnews.com/2018/01/09/clinical-trials-reporting-nih/. Accessed 8 April 2019.