Peer Review: A Flawed Process at the Heart of Science and Journals

16
ByCrossFitFebruary 4, 2019

Richard Smith, former editor of the BMJ and chief executive of the BMJ Publishing Group, presents an insightful critique of the broken peer-review process.

For years, peer review has been held up as a measure of quality and authority in publication. Yet those most familiar with the subject—such as Smith—know peer review is a slow, ineffective, and costly process characterized by bias and abuse, and peer-reviewed articles are often no more accurate or valid than papers that have not been reviewed at all.

As Smith notes, “Peer review is a flawed process, full of easily identified defects with little evidence that it works. Nevertheless, it is likely to remain central to science and journals because there is no obvious alternative, and scientists and editors have a continuing belief in peer review. How odd that science should be rooted in belief.”

Comments on Peer Review: A Flawed Process at the Heart of Science and Journals

16 Comments

Comment thread URL copied!
Back to 190205
Matthieu Dubreucq
November 13th, 2019 at 2:48 pm
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

I wonder what could replace Peer Review. That would be an interesting subject. Maybe we should also start with increasing the quality of studies and diminishing the number of them. No need to have 1 000 wrong studies when you can have 5 good ones.

Comment URL copied!
Mary Dan Eades
February 7th, 2019 at 5:15 pm
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

The peer review process reminds me of a discussion from years ago at a media and marketing symposium we attended. The speaker, a former Pulitzer-nominated Wall Street Journal investigative reporter, asked us: 'What is news?' And attendees variously responded: a major transportation accident, the death of a public figure, the passage of a major piece of legislation, etc. One after another opinions of what might be considered 'news' arose from the group. The speaker just shook his head and said, "News is whatever the editors/publishers/producers of newspapers, magazines, television say it is." Meaning, of course, that what gets published--ie makes the news--is utterly subjective.


Nowhere does this seem more true than in academic publishing as is so astutely pointed out in Richard Smith's piece. Low carb nutritional researchers (or those producing research saying anything that even seemed to suggest that eating more fat was a good thing) labored to get their work published, chiefly because the journal editorial boards were in low-fat lockstep ideologically. And so arose the Catch-22 of low-carb research: the demand for studies that proved that position and the refusal of major journals to publish them even if they did. And from that logjam arose the founding of an organization without the prevailing biases, the Nutrition and Metabolism Society, and its associated journal. At last, there was a peer-reviewed nutritional outlet for low-carb . In recent years, the tide has begun to turn and more and more and more good low-carb research is finally being published, but it hasn't been without a struggle.


Journalist and the public invest tremendous gravitas in the peer-review process. It The review process is based on 'trust' that those submitting their research conducted it with integrity, are reporting their results accurately and honestly (both the good and the bad, the positive and the negative) and disclosing their conflicts fully. And, too often, that isn't what happens.


The peer review process, as so clearly detailed by Smith, is flawed, perhaps fatally so. And worse, it fails to accomplish the noble end intent for which it was set forth -- to weed out bad, fraudulent, corrupted, weak science. But that isn't what it does.

Comment URL copied!
John Lee
February 5th, 2019 at 9:55 pm
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

Peer review is a result of what science is at it's core - a free market of ideas backed by evidence. A scientist proposes something that furthers our understanding of the world and presents it as a paper. Peer review is the first step of critiquing and insuring that the science is correct. Yes, the 3-4 reviewers may not find all the flaws in a paper before it's published. But afterwards it's open to discussion and refutation by their peers. Sometimes papers still have errors after publication, but eventually those papers lose significance. Only if an idea has stood the test of time, after weathering assault by competing ideas, is it considered scientific fact.


All scientists understand that there are flaws in the peer review system. No reputable scientist bases their assumptions on a single paper. Experts look at the entire body of work, with confirmation by multiple sources, before making a conclusion. That's why science continues to progress despite flaws in the peer review system.


Also your articles against peer review just suggest another form of peer review. For example, the Gowers suggests that articles are put directly online and are open for comments before publication. I find it ironic that comments are heavily curated on this website and it's articles are not open to criticism.


To say that "most published research findings are false", or that peer review "is a flawed process at the heart of science" seems to be the start of an effort to discredit science. Presenting just one side of the story, in the form of 10 and 15 year old opinion articles, reeks of anti-science agenda.

Comment URL copied!
Tyler Hass
February 5th, 2019 at 11:05 pm

If only science was confined to experts who look at the entire body of work. Unfortunately, if a shoddy piece of work gets press coverage or a retweet from a Kardashian, then it's elevated to scientific law in the eyes of the public. Health and nutrition science is notoriously error prone and it also gets the most press coverage.

A single paper that gains traction can take years to be refuted, even when counter evidence has been presented. Just look at the impact Ancel Keys' study had. The vast majority of people today believe cholesterol is strongly linked to heart disease. Low fat food-like products have government approval to place a "heart healthy" sticker on their packaging.

I'm surprised to hear people saying that CrossFit is anti-science. These articles that have been posted over the past month are anything but. The authors range from physicians who deeply care about their patients to researchers who care about the integrity of the scientific process. If there's an ulterior motive here, I'm not seeing it.

Comment URL copied!
Shakha Gillin
February 6th, 2019 at 12:38 am

John, I appreciate your discussion.


I don’t agree that papers that have errors lose significance over time. We have a measles outbreak as we speak, fueled by Dr Wakefield’s retracted paper from 1998. Damage was done, and the retraction didn’t change the public perception, 20 years later.


It’s not anti-science to discuss flaws of the peer review process, which is not science to begin with. We need to separate that which is ingrained in the culture of medicine with what defines science. For me, I see the goal as going back to real science so we can make some headway.

Comment URL copied!
Phil Wernette
February 6th, 2019 at 2:33 pm

John, I agree with your characterization and am pleased to see that I am not the only one concerned. Some are arguing to "going back to real science" as though the process of research and science has changed. Science is still science and remains founded in a process of 'searching" for answers and 're-searching' to validate or refute the previous answers. What about science, itself, changed? If anything, the scientific process has improved significantly from decades ago when journals did not require or even request funding sources or potential conflicts of interest be identified.


The issue with the series of recent articles by CF HQ is really cherry picking studies over a decade old to discredit most published research. And I would agree that it really does reek of an underlying anti-science view, which is particularly troubling given CF's global coverage and audience and the fervor for which the CF community follows HQ. It seems prudent to consider the potential broader impacts on community health before sharing any article, as some posts may have the potential to do more harm than benefit.

Comment URL copied!
Greg Glassman
February 12th, 2019 at 2:35 pm

John,


I disagree with most of what you’re saying but understand your attempt to explain science. Your view, while popular, is not even close to correct.


Science is not “at it’s (sic) core - a free market of ideas backed by evidence.” It’s the activity of producing theories with better than chance predictive value. The method by which this is done is "the scientific method". Peer review is NOT a part of the Scientific method. Scientific theories map a fact to a future fact, both determined empirically, establishing a causal relationship, attached to “the reals” with a measurement and degree of measurement error. That’s how we put man on the moon, your TV was invented, and medicine nearly became a science.


That, Sir, is what modern science is. You’re describing “post modern science” or “consensus science”. It exchanges the fruit of modern science, theories with better than chance predictive value, with consensus, peer review, etc.


This has created a climate that guaranteed innumeracy, scientific misconduct, corruption, and the replication crisis. (And ultimately to a deadly, tragic, non-response to chronic disease where the cause of chronic disease has been offered to the public as a preventative.)


What do you mean by “heavily curated”? I can’t find meaning. Are you referring to the comments and posters removed for being rude or launching ad hominem attacks? Every publishing effort is “heavily curated”. The PRJ’s you trust are “heavily curated”, no?


You’re joining the chorus that things that science becomes stale somehow after some number of years. Isn’t that absurd if you just think about it? The stale science that we have featured includes work that is STILL cited as the backbone for the diet-heart hypothesis. When Framingham is no longer cited, its obvious limitations won’t be worth mentioning other than as an historical oddity.


Finally, your comment that “this website and it’s (sic) articles are not open to criticism” can only be made if you hadn’t looked at the site, read your own comments, and caught on to what we are actually hoping happens here. It’s an amazing statement from that regard alone.


Thank you.

Comment URL copied!
Greg Glassman
February 12th, 2019 at 2:40 pm

John,


I disagree with most of what you’re saying but understand your attempt to explain science. Your view, while popular, is not even close to correct.


Science is not “at it’s (sic) core - a free market of ideas backed by evidence.” It’s the activity of producing theories with better than chance predictive value. The method by which this is done is "the scientific method". Peer review is NOT a part of the Scientific method. Scientific theories map a fact to a future fact, both determined empirically, establishing a causal relationship, attached to “the reals” with a measurement and degree of measurement error. That’s how we put man on the moon, your TV was invented, and medicine nearly became a science.


That, Sir, is what modern science is. You’re describing “post modern science” or “consensus science”. It exchanges the fruit of modern science, theories with better than chance predictive value, with consensus, peer review, etc.


This has created a climate that guaranteed innumeracy, scientific misconduct, corruption, and the replication crisis. (And ultimately to a deadly, tragic, non-response to chronic disease where the cause of chronic disease has been offered to the public as a preventative.)


What do you mean by “heavily curated”? I can’t find meaning. Are you referring to the comments and posters removed for being rude or launching ad hominem attacks? Every publishing effort is “heavily curated”. The PRJ’s you trust are “heavily curated”, no?


You’re joining the chorus that things that science becomes stale somehow after some number of years. Isn’t that absurd if you just think about it? The stale science that we have featured includes work that is STILL cited as the backbone for the diet-heart hypothesis. When Framingham is no longer cited, its obvious limitations won’t be worth mentioning other than as an historical oddity.


Finally, your comment that “this website and it’s (sic) articles are not open to criticism” can only be made if you hadn’t looked at the site, read your own comments, and caught on to what we are actually hoping happens here. It’s an amazing statement from that regard alone.


Thank you.

Comment URL copied!
Greg Glassman
February 12th, 2019 at 2:41 pm

John,


I disagree with most of what you’re saying but understand your attempt to explain science. Your view, while popular, is not even close to correct.


Science is not “at it’s (sic) core - a free market of ideas backed by evidence.” It’s the activity of producing theories with better than chance predictive value. The method by which this is done is "the scientific method." Peer review is NOT a part of the scientific method. Scientific theories map a fact to a future fact, both determined empirically, establishing a causal relationship, attached to “the reals” with a measurement and degree of measurement error. That’s how we put man on the moon, your TV was invented, and medicine nearly became a science.


That, Sir, is what modern science is. You’re describing “post modern science” or “consensus science." It exchanges the fruit of modern science, theories with better than chance predictive value, with consensus, peer review, etc.


This has created a climate that guarantees innumeracy, scientific misconduct, corruption, and the replication crisis. (And ultimately to a deadly, tragic, non-response to chronic disease where the cause of chronic disease has been offered to the public as a preventative.)


What do you mean by “heavily curated”? I can’t find meaning. Are you referring to the comments and posters removed for being rude or launching ad hominem attacks? Every publishing effort is “heavily curated”. The PRJ’s you trust are “heavily curated”, no?


You’re joining the chorus that thinks that science becomes stale somehow after some number of years. Isn’t that absurd if you just think about it? The stale science that we have featured includes work that is STILL cited as the backbone for the diet-heart hypothesis. When Framingham is no longer cited, its obvious limitations won’t be worth mentioning other than as an historical oddity.


Finally, your comment that “this website and it’s (sic) articles are not open to criticism” can only be made if you hadn’t looked at the site, read your own comments, and caught on to what we are actually hoping happens here. It’s an amazing statement from that regard alone.


Thank you.

Comment URL copied!
Greg Glassman
February 12th, 2019 at 2:44 pm

Phil,


Yes, you got it right. We’d like to “go back to real science”. Science that produces theories with better than chance predictive value. That is what is called “modern science”. It begins with Aristotle, is honed sharp by Bacon, and has produced all of the technical marvels of our modern age. It got replaced with consensus science, or post modern science. There, the goal is not to produce theories with better than chance predictive value but to form consensus. Peer-reviewed journals showcase that consensus and nothing more. The essential measurement in consensus science is voting. There is no voting in modern science. I’ve become a tad redundant to drive home a very simple and vital notion.


Post modern science lives in the universities and not in industry. Industry needs its theories to work, academia needs consensus instead.


I’d like to explain to both you and John Lee the essentialness of empiricism in science. Theories have an obligation to comport with empirical reality. One of our physicians came to a moment where she “had to follow her training and education or her lying eyes.” There’s only one path here if you are going to call your approach scientific. The modern scientist would go with “lying eyes” the post modern, academic, consensus science will go with the training and education every time. Much of what is wrong in medicine, public health, and fitness training owes to this colossal case of the Emperor’s New Clothing.


Phil, you summed with “The issue with the series of articles by CF HQ is really cherry picking studies over a decade old to discredit most published research.” I want to respectfully suggest that the issues are yours and stem from not having an understanding of what science is. You’re not alone. We are going to “cherry pick” and “heavily” curate articles forging our way to a legitimate science that works.


I’m going to give a spoiler alert for those looking for only “fresh” stuff. Math, logic and the scientific method don’t need fresh resources, and correcting a botched scientific legacy will also require looking at some old stuff. Think forensics, not Kardashians.

Comment URL copied!
Steve Lewis
February 25th, 2019 at 3:07 am

That is exactly how the FDA (Food and Drug Administration) operates. As stated by several clients of mine who are on the FDA Board, on the outside it is a peer review, but at the end of the day it is nothing more than a consensus.

Comment URL copied!
Phil Wernette
February 5th, 2019 at 6:29 pm
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

The peer review process is certainly not perfect, nor is the publishing process as a whole. It can get political and biased in a hurry, which can bury important work and findings. Depending on the journal, the specific editor, the reviewers, and other factors, the quality of peer review can vary substantially. I agree with Smith that it's not a perfect process and I would like to see a viable alternative that serves the same idealogical purpose. However, I would propose that there is some (even if anecdotal) evidence that the process can work as intentioned. Many articles submitted for publication have significant gaps in their research, neglect alternative explanations for a finding, are incoherent, or have other glaring issues (including potential funding conflicts of interest). Without peer review, these articles would be given the same weight as research that is sound and rigorous. So, is it a perfect system? No, but it can function effectively, depending on a variety of factors.


In contrast to many traditional journals, open-access journals and digital only journals are helping change this process for the better. From my experience, open-access and digital only journals tend to be quicker and less biased with their review process because all comments are open to the public. This may temper some of the bias that might otherwise creep into the review process. The open review system can actually provide very high quality feedback that is essential to improving the research and the article. The new challenge with these alternative publishing outlets can be determining which journals are legitimate and not predatory, which is a completely different discussion.

Comment URL copied!
Melissa Yinger
February 5th, 2019 at 5:34 am
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

The study Smith cites for his reference to discrimination against women in the peer review process, Christine Wennerí¥s and Agnes Wold's "Nepotism and sexism in peer-review," is worth a read as well. Much like Smith, they argue that the "policy of secrecy" habituated by the peer review process enables abuse and admits bias. They encountered a number of hurdles when trying to get data related to peer review because of this policy of secrecy. And because the data they were able to acquire through Sweden's Freedom of the Press Acts demonstrated that secrecy in the peer review process was covering over bias of a considerable magnitude, they argue that "the credibility of the academic system will be undermined in the eyes of the public if it

does not allow a scientific evaluation of its own scientific evaluation system."

Comment URL copied!
Clarke Read
February 5th, 2019 at 5:28 am
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

"An alternative system would almost certainly not be perfect, but to insist on perfection, given the imperfections of the current system, is nothing but status quo bias. To guard against this, imagine that an alternative system were fully established and see whether you can mount a convincing argument for switching to what we have now, where all the valuable commentary would be hidden away and we would have to pay large sums of money to read each other’s writings. You would be laughed out of court."


Gowers has written one of the most concise and precise reviews I’ve seen of the intention of peer review, the reality, and the gap between the two. Peer review serves as either preliminary due diligence for readers, a tool for curation, or a means to improve the quality of the field - and yet it is questionable if not outright lacking in each of these areas. The implications are significant, as peer review is arguably (i.e., I’d argue) one of the only meaningful distinctions between formal academic research and simply gathering evidence via a scientific method. The former is given a level of significance far beyond the latter. (Just think of the emphasis journalists place on the fact certain pieces of evidence are peer-reviewed - either directly or by implication through noting their place in an academic journal.) But if the formal peer review process does not actually ensure or improve the quality of the evidence subjected to it, we are over-valuing some forms of evidence and under-valuing others.


Gowers generally highlights the problem; others like Aaron Carroll over at NYT (link below) have reviewed solutions and I know plenty of journals have been experimenting. I doubt this will be the last time this topic is covered here on Crossfit.com, and I’m looking forward to exploring this topic further. A system that better serve the core objectives Gowers outlines could improve accountability and rigor throughout academia, and so help us get more out of our research investments.


https://www.nytimes.com/2018/11/05/upshot/peer-review-the-worst-way-to-judge-research-except-for-all-the-others.html

Comment URL copied!
Jeff Glassman
February 10th, 2019 at 4:24 pm

Clarke,

Interesting that you would post about Gowers. By the way, his article can be found here:


https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/


Gowers first sentence reads,

“In 1998 the Lancet, one of Elsevier’s most

prestigious journals, published a paper by Andrew

Wakefield and twelve colleagues that suggested a

link between the MMR vaccine and autism.”


Later he adds,

“It is not easy to have a paper published in

the Lancet, so Wakefield’s paper presumably

underwent a stringent process of peer review.”


Modern Science (Bacon (1620) to the present) may *assume* (a contingency), but never *presume* (a belief). Post Modern Science (Popper (1934/1959) to the present in academia) may presume away to its collective heart’s content, but apparently the *stringent process* was one presumption too many.


Two years after publishing Wakefield, Richard Horton, MB ChB, Lancet Editor-in-Chief, 1995 — present, had this to say:


“Peer review as a reliable technique for

assessing the validity of scientific data is surely

discredited. ¶ The mistake, of course, is to have

thought that peer review was any more than a

crude means of discovering the acceptability

- not the validity - of a new finding. Editors and

scientists alike insist on the pivotal importance

of peer review. We portray peer review to the

public as a quasi-sacred process that helps to

make science our most objective truth teller. But

we know that the system of peer review is biased,

unjust, unaccountable, incomplete, easily fixed

[jiggered, not repaired], often insulting, usually

ignorant, occasionally foolish, and frequently wrong.

Horton, R.C., *Genetically modified food: consternation,

confusion, and crack-up*, Med.J.Aust. 172(4),

2/21/2000, 148-9.


Read Controversies : Denial of retraction of discredited vaccine at Wikipedia >> Richard Horton,

https://en.wikipedia.org/wiki/Richard%5fHorton%5f(editor) to see what Horton unleashed.


*Assessing the validity of scientific data?* Really? Peer reviewers are not known to ever provide quality control over data. It’s supposed to be quality control over scientific propositions, known better as models. Peer review is raw, biased opinion (Horton), one of Popper’s triad of intersubjective tests with which he replaced most of objectivity (he deleted others in different ways) in the propositions of Modern Science. Popper intended it to free science from its annoying, successful objectivity, and re-open the door to his cherished metaphysics. Popper was competing with the Vienna Circle by claiming to have solved the problem it had set for itself (and in the end failing), all without the presence Popper.


What exactly is peer review supposed to accomplish, if not the skills Horton attributes to it? Gowers says it's supposed to (1) ensure reliability, (2) select the most important articles for publication, and (3) provide feedback. However, the reviewers are anonymous, and their comments secret, so it cannot be known to do any of those things. It’s a secret gatekeeping function for the benefit of the editors, the keepers of the dogma (Horton).


Gowers (2017) sports this subtitle: “Timothy Gowers on the alternatives to formal peer review”. But Gowers only manages to imagine what an alternative might be. He asks, “What would the world be like without formal peer review?” He provides a hint in his quest - he criticizes peer review for this apparent failure:


>>Nor does formal peer review seem to manage

very well to stop wrong ideas from spreading outside

academia. Climate change deniers are not put off by their

lack of representation in respectable academic journals.


Gowers has this perfectly inside-out, wrong on every count. Climatologists of the Climate Change persuasion only publish in approved journals, and those journals, in fine Hortonesque fashion, only publish Climate Change dogma, never serious criticism. Climate Change is peer reviewed as a *movement*, not as a science.


Early on, the Movement won the argument by commandeering the vocabulary, formally changing *Anthropogenic Global Warming* first to *anthropogenic climate change*, and then abbreviating the latter to *Climate Change*. Where the Movement pretended to show the fingerprint of humans on climate observations, it did so by outright fraud in the form of doctored charts (*chart junk* in the science vocabulary).


The Movement was far from finished. It borrowed from religion to label its skeptics, a scientific virtue normally, as *deniers*. Even before that, when its models early on proved incompetent at predicting temperature, they changed Global Warming to Climate Change. Nevertheless, Climate Change is an excellent paradigm to answer the question Gowers poses: how might scientific papers work absent “formal peer review”.


A key player in the evolution of the Climate Change Movement was Guy Callendar. In 1938 he cobbled together the physics of a half dozen investigators to publish *“The Artificial* [manmade] *Production of Carbon Dioxide and Its Influence on Temperature”*. His model was briefly called *The Callendar Effect*, and later changed by the Movement to *The Greenhouse Effect*, preferring a misnomer to giving credit to a failed work by a lowly steam engineer.


Today, scientific papers regularly sport a Discussion section wherein the author provides a narrative for his study. Reviewers are anonymous, and their comments secret throughout the process. The result has been a crisis in academic publishing. Instead, Callendar (1938) has a Discussion section containing the commentary of six identified reviewers, paraphrased in the third person by the editor, and followed by the author’s responses. The first reviewer, Sir George Simpson, head of the Met Office, snarkily remarked, among other things,


“ ... it was not sufficiently realized by

non-meteorologists who came for the first time to help

the Society in its study, that it was impossible to solve the

problem of the temperature distribution in the

atmosphere by working out the radiation ... “


While no facts support the conjecture of even the existence of Anthropogenic Global Warming (AGW; it remains unobservable and hence unmeasurable), the pioneering paper by Callendar, from a field exemplified by Gowers, stands as an example of how peer review used to be, pre-Popper, and successful. Callendar (1938) was published, as it should have been, warts and all.


Publication in the past is a candidate for where professional publication might go, and without anonymous peer review.

Comment URL copied!
Katina Thornton
February 5th, 2019 at 2:39 am
Commented on: Peer Review: A Flawed Process at the Heart of Science and Journals

Just as "what is fitness" brought forth a myriad of responses, none scientific until Coach Glassman, so does "what is peer review" evoke a number of responses, again none scientific.

Richard Smith said it best: "How odd that science should be rooted in belief.”

Comment URL copied!