Thursday

191128

Workout of the Day

5

Rest Day

Post thoughts to comments.

Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

162

“American taxpayers spend $30 billion annually funding biomedical research. By some estimates, half of the results from these studies can’t be replicated elsewhere — the science is simply wrong. Often, research institutes and academia emphasize publishing results over getting the right answers, incentivizing poor experimental design, improper methods, and sloppy statistics. Bad science doesn’t just hold back medical progress, it can sign the equivalent of a death sentence. ... In Rigor Mortis, award-winning science journalist Richard F. Harris reveals these urgent issues with vivid anecdotes, personal stories, and interviews with the nation’s top biomedical researchers. We need to fix our dysfunctional biomedical system — now.”

Read MoreRigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Comments on 191128

167 Comments

Comment thread URL copied!
Todd Nief
December 31st, 2021 at 12:26 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis catalogs many of the known failure modes for scientific research: p-hacking, sloppy statistical methods, publication bias, poor experimental design, a cutthroat prestige economy in academia, a constant need for more funding, and, of course, outright fraud.


The diagnosis is one of rigor mortis: entrenched bureaucracies, antiquated funding models, and self-interested scientists easily corruptible by big money spend their careers churning out shoddy publications.


While I'm in agreement with most of these criticisms, I have concerns about how this information is received by the CrossFit community — especially based upon the other comments on this post. The response seems to be something to the effect of: "See, corruption has rendered these institutions worthless. Instead, I'll trust my gut and disregard experts since they're just pawns in a system lining the pockets of the Sacklers...or maybe even the Rothschilds." This take certainly aligns with the priors of the CrossFit community, where the exercise science literature has historically focused on legible experiments of EMG data and single-joint exercises and "in the trenches" coaches have developed far more effective methods through trial and error, mentorship, and internet bro science.


Something more pernicious is at play though in the discrediting of experts I see here. This phenomenon is well-outlined in Martin Gurri's "The Revolt of the Public" and is proudly on display in this comments section. Elite failure is seen as irrevocable and irredeemable proof of corruption and incompetence. Rather than demanding that our institutions live up to the ideals that we would expect, we instead demand that we flip the whole scientific table over. For an intuition of this dynamic at play, try listening to daytime sports talk radio. Enraged fans scream into their phones about how a professional athlete is completely incompetent and not worth a fraction of the millions he's paid because he dropped a pass, missed a field goal, or didn't play through injury. The rage is palpable, and the polemical couch potato feels entitled to belittle *one of the literal best athletes in the entire world* because *he saw him drop the catchable pass with his own eyes.*


A more nuanced treatment of the sclerosis, corruption, and in-fighting in academia might instead focus on the lack of low-hanging fruit in medical research for the last several decades. To be clear, all of the problems highlighted have *always existed in research.* However, there were fecund planes of new discoveries in nearly every direction earlier in the 20th century. As such, the most competent research were actually discovering groundbreaking truths rather than fighting for their share of a limited pie. The institutional rigor mortis we see is more likely a symptom of fewer opportunities for new discovery — not necessarily the root cause.


I will add that this book's publishing time was unfortunate, as it came just before we are starting to see huge breakthroughs in computational biology (protein folding problem, anyone?), gene editing, machine learning in drug discovery, and mRNA vaccine technology. Additionally, we've seen new funding models like those pioneered by Patrick Collison (of Stripe) and Tyler Cowen (of Marginal Revolution) focusing on Covid research through their Fast Grants project. If my thesis is correct, this will realign the incentives in research and we will see more energy focused on producing real world, repeatable results. Any human institution will be prone to corruption and some percentage of individuals will always seek to elevate their status at the expense of the organization. However, when there is *actually something to work on,* we will hopefully see organizational cultures realign to point in that direction.


To be clear, the problems mentioned in this book are real and they are severe. Better statistical methods are not yet widely adopted, and few (if any) disciplines have yet come to terms with the replication crisis. The rot is almost certainly too severe to save in some universities, government agencies, and corporations. Ideally, they are outcompeted by faster, more effective, and more rigorous organizations.

Comment URL copied!
Zachary Forrest
December 8th, 2021 at 7:20 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Few books for me have called into scrutiny some personal beliefs I've long held to be true...but Rigor Mortis did so in a roundabout way that manifested as a particularly enjoyable book. As humans, we are seeking Truth on many different levels and what we are willing to accept is usually anything but...it comes down to not only our own trust levels, but also...Who Marketed Better. Rigor Mortis teaches us that we all need more analytical criticism when deciding where to place our trust.


As previous comments spotlight: A "one size fits all" approach to scientific studies isn't feasible...either because of logistics or variables that aren't able to be controlled. The issue then becomes "How do we mitigate as much of the unknown as possible and ensure adherence to a given approach?" Do the amount of resources needed to surmount such a task actually provide ROI on the result? It would seem that each study would need this evaluation done on it's own premise, but even THAT could be an entire field of study that requires further exploration.


In my opinion, the biggest takeaway a reader could glean from Harris is also the overarching point of the book: bad science happens...it happens often...and we don't even realize it sometimes. Due to ignorance, corruption,

Comment URL copied!
Bill Grundler
October 5th, 2021 at 1:48 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

When you think about how lucky we are to be able to have scientists whose pure drive in life is to find answers to our problem, it gives you a very comforting feeling. These folks with special skills and knowledge are working diligently to solve these massive health problems for us…….or at least thats how we used to feel.


The book goes through biochemical research and examines how many of our past and current findings are, and have been, false. The author stated that approximately only 15% of finding are actually true!!  But why?? Competition always will drive a higher level of pursuit. The problem is when money gets involved, scientists and research companies need funding to do their work. To get the money you need to have successful results. The drive to get these results means that they might be falsified, or modified. Big companies need more money and more money means more success needed and more spectacular findings. Pushing this opens the doors even more to falsifying more research. It ends up being a vicious circle! It’s so difficult to read think how many of our “truths” are actually not true at all but rather narratives that were created to show great success by this or that company, only in the hopes of getting more money. 


This forces me to question everything that we are told will help us. The problem is that the questioning doesn’t always give you answers. What information is actually correct?? So instead of gaining a better understanding on what I’m trying to figure out, I get lost in false truths. UGH!!  


I don’t fault the scientists. I think there is a moral responsibility to try to do the right thing, and I think most have this morality and general drive to find the REAL truth. They need money to do their thing and we should spend much to gain much. It seems to me the only real way our scientists can truly find truths out there is when they are able to research, find, fail, retest, fail, find, etc until the “end” of the question, without the controlling shackles of money and big orgs directing them. I don’t know what the standards to control this would look like but it seems there needs to be something so that we aren’t wasting money and lives for the sake of more grants.  


Guess I’ll get back to eating my chicken and broccoli and do my constantly varied functional movements done at high intensity because at least I KNOW that works!

Comment URL copied!
Carmen Casteling
September 16th, 2021 at 5:10 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis was a great read. Even though it was daunting to read about how scientific research is cherry picked and researchers choose to show ‘best results’ which falsely advertises the experiment. It reminds me of the importance of consistently improving myself and my own critical thinking skills whilst still staying humble. It is important that one consistently keeps learning, still fact checking and looking around to see what others are doing/saying and remaining unbiased. 


I was shocked to read about the medical field not making any improvements in the last 40 years and the inaccuracy of scientific journals and the overall lack of quality in scientific research. If we cannot trust medical journals, what can we trust? 


In my opinion, there is too much room for error with animal testing in connection with human diseases. Not only are mice and rats not similar to humans but the environment and other instances just don’t help the scientists make the right connection for their studies. It states, “male animals often used in the studies may not be a good substitute for an elderly human being having a stroke” (p. 76) as well as  “improvements won’t help in the many instances when animals are poor stand-ins for human disease” (p. 76).


On another note, I am a big animal lover so reading about what all the animal testing pharmaceutical companies need to do was hard. I was happy to read, “Pharmaceutical companies are already starting to use chips in place of mice for experiments” (p. 87) which will not only steer scientists away from using animals but also prepare them better to use similar testing to humans in the future. 


I understand for scientists to be able to discover breakthroughs with medication and cure diseases thorough research with plenty of funding needs to be available, however reading this book has opened my eyes to seeing that that research needs to be more thorough than what it currently is to allow tests to be successful. There should be no rush to publish something that cannot be reproduced with similar results or that hasn’t not been tested for the long term success rate. 


No one would want inhumane testing on people before the testing stages has been explored thoroughly and is reliable to know that humans will be safe. This statement, “Cleveland Clinic would not comment or talk about its processes for protecting the patients in these experiments,” (pp. 105-106), does not reassure anyone that testing is always safe and successful. 


Overall, the takeaway from this book was that the professionals need to get results fast. Speeding up the research to get results often produces bad results. This is also true for health and wellness. Health and wellness is a lifelong journey and should not be marketed as a quick fix, rather a lifestyle change. 


Comment URL copied!
Krista Fudge
September 3rd, 2021 at 8:07 pm
Commented on: 191128

Rigor Mortis by Richard Harris was absolutely alarming, and makes me rethink everything I've ever learned in the past. Everything I believe that is based on "medical research" has lost its luster for me.

And this book was very timely with the current state of the world, and all of the studies and "research" being done for COVID.


Richard Harris uncovers "sloppy science" procedures, and explains that over half of the studies being done cannot even be replicated due to poor experimental designs and biases, pressures to get published, pressures to get funding, and many more reasons. I found it very interesting when Harris explains that the issue is not all due to sloppy scientific techniques. Scientists truly do want to produce findings that are beneficial to society. Harris writes that funding is scarce and the pressure to publish in high-impact journals is intense. As a result, scientists prioritize studies that are likely to get the most attention over less sexy but perhaps more beneficial work. Harris explains that the pressure to publish and receive grant money drives scientists to run experiments in a way that undermine the reliability of their work. Sometimes scientists will exaggerate results or ignore data that doesn't fit their theory in order to 'publish' the results they want.


Clearly, this is an issue that will take a while to fix, but hopefully Harris' book encourages every scientist to scrutinize her experimental design and encourage other scientists to try to reproduce findings. In the meantime, I'll be leery of anything that claims to be backed by medical research.

Comment URL copied!
Matthias Turner
June 30th, 2021 at 2:30 am
Commented on: 191128

The Book Rigor Mortis by Richard Harris is a real "wake-up" book!

To be honest it was actually quite scary to see how little scientific evidence is repeatable and in turn, potentially not at all true… Especially when we live in a world the is so driven by data! 


One of the first big things that stood out was the amount of money that is pumped into science… Typically with the desire of founding out something that the funder is wanting to find out. With the scientific field becoming even more cutthroat it’s almost no surprise that at times falsified data/theories are given.  


Unfortunately though, at times scientists seem childish… or more so how science as a whole seems to go about their findings is childish. I once talked to a scientist who told me the whole point of his job was to discredit other scientists finding. This comes across in the book. As well as a lack of wanting to share data (right or wrong) to help the greater good of science to take steps further forward together! 


It seems like a simple have you followed these steps system implementation could help with a lot of wasted trials, studies and findings… How do you govern a body that’s not open to it though?


It seems that with the implementation of new technology we can expect great things… But also greater mistakes being made. The perfect example of this is the A.I bot in which they fed the 1000 photos of static into. The bot was programmed to find a photo of Einstein… It claimed to have found a photo and spat out the clear picture of Einstein even though. It wasn’t fed into the machine! 


This book has changed the way I view scientific backing… It really seems that the scientific world, unfortunately, lacks integrity! 


Regarding CF I think it shows that a trial and error approach to coaching at times is the best method. Everybody is slightly different and the coaching cue that works best for an individual is the one that encourages the correct change you were aiming for. If at first, you don’t succeed, try and try and try again!

Comment URL copied!
David Whitty
June 28th, 2021 at 1:17 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

“Rigor Mortis” by Richard Harris paints a disturbing reality of the squandering of billions in the pursuit of self promotion and security for the biomedical research community. Sloppy science and acceptance of a study as ‘fact’ without seeing if it something that is reproducible could not be more clearly illustrated as was done by Glen Begley when it was revealed that one study was cited 2000 times by other researchers… without actually validating the underlying results.”(p 12). 


The need to examine the bias of the researcher is left as an after thought that may only be mentioned in a line of no financial funding from this or that type organization. 


Working in health care as well as fitness the need to be transparent when providing a plan of management that is reasonable and evidence informed is a reality of my everyday. There is nothing wrong with telling a client or patient that we do not know based on current knowledge. It does not seem unreasonable to ask the same of researchers to own up to uncertainty when finding are questionable which they mostly are!


I actively support research financially and will continue to do so but am thankful when I see transparency in the granting of research dollars. Coordination of research objectives by funding organizations is necessary but the researchers themselves must be left to journey where the data actually leads without interference. 


When the statistics of Mendel’s pea pods are examined the numbers are too good to be true when considering genetic drift! Was the monk a poor note keeper? Was his work intentionally leaving out examples that did not find the hypothesis? 

In summary my hope is that rationalization of efforts, sharing of resources and a shift to for the better of humanity will prevail over the almighty dollar. 

Comment URL copied!
Deane Ketzner
May 28th, 2021 at 9:18 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions


Rigor Mortis depicted just how skewed my perception of the biomedical field has always been. As somebody who enjoys perusing articles in journals, it’s a disturbing reality that almost the majority of published works have not been proven beyond reasonable doubt, and are incapable of being reproduced the majority of the time. In fact, it seems as though conclusions drawn from scientific experiments would fail to even be proven effective on a balance of probabilities. 


The unconscious bias that scientists apply when performing research experiments, and the pressure they face for failed experiments, forces scientists to act in ways that not only halt the progression of biomedical research, but also causes them to question their capabilities and integrity. There is such a big focus on scientific development and producing results, when in fact, scientists should be taking a step back and analysing their results objectively in order to ascertain why an experiment of theirs was unsuccessful, in order develop something better. 


An eerie parallel can be drawn between Rigor Mortis and Uffe Ravnskov’s Cholesterol Myths – both authors uncover ugly truths plaguing the biomedical faculty. The most alarming being the rate at which scientists rely on, and apply results from animal experiments, to humans. Although it is not necessarily the fault of scientists, but rather those who set the threshold in order for a study to be published and to be of value, scientists should act with integrity. There should also be a certain level of accountability, and measures to reprimand scientists that deviate from the standard. 


This brings us to the next issue, which is the duty of disclosure. Researchers within the biomedical field, and in any profession in general, should have a duty to disclose any findings which are unfavourable, or which caused their research to fail, in order to prevent further mistakes in the future, and to allow greater progression. Therefore, not only are scientists getting away with bad science because they do not have a duty to disclose their analysis, they are hampering scientific advancement in the biomedical sphere. 


Scientists do not have bad intentions. Their world is simply overwhelmed by structural and financial issues, making it a dog-eat-dog world. Without publishing papers, they do not qualify for funding. Therefore, scientists publish papers, even if they have very little scientific value, in order to attain funding, which allows them to continue researching. It’s a vicious circle that needs to be addressed in order to attain meaningful outcomes in the future.


Rigor Mortis has taught me to question everything, because the lack of improvement in the medical field is not due to impossibility. Financial and political factors play a conspicuous role in biomedicine, and are clear underlying factors resulting in the inactivity thereof. Therefore, until these external issues are addressed, scientific stagnation is guaranteed.


Comment URL copied!
errol clark
May 17th, 2021 at 6:44 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis was an intriguing read. Overall, it reminded me of the importance of consistently improving my critical thinking skills while remaining humbled by the fact that science still is in its infancy with regards to biomedical science and health.


I remain optimistic that the truth will emerge with time. 


I remain pessimistic that the major diseases that plague us will be “cured” within our lifetime. 


The book strengthened my bias towards the importance of preventive healthcare and our role as health coaches. During our lifetime our best bet is to prevent disease in the first place. Lifestyle choices, exercise, nourishment, and stress reduction are paramount. 


Our skills to inspire them in others are crucial. 


I was shocked by the reproducibility issues at large. The suggestive data is staggering and certainly warrants the emerging field of meta-research. Unfortunately, it also left me feeling a little helpless. I’ve long sought after credible information in regards to health. This book reaffirms something I’ve suspected for years… much of biomedical science and the health sciences in general cannot not be fully trusted, no matter what journal they are referenced from. 


It pains me to see smart individuals like Gary Tuabes and Stephan Guyenet argue contrarian yet seemingly valid opinions, both of which can be backed by a swarm of published articles that I have little chance of reviewing within my lifetime (Have a listen if you haven’t already - Joe Rogan #1267 - Gary Taubes & Stephan Guyenet). These whitepaper battles leave me feeling further confused at times. I’m over 15 years into my journey as a health coach and still climbing the mountain from the bottom of the curve on the Dunning-Kruger effect, consistently humbled by biology and the human body. 


Until science catches up (again not likely within my lifetime) I will remain steadfast in my coaching approach; thorough assessment, personalized prescriptions, and most importantly, reassessment of what’s working and what’s not, altering my approaches along the way.     


Other key takeaways from the book: 


1) Just because something is published in a medical journal, certainly doesn't give it instant credibility. 

2) There is supporting evidence that only a fraction of studies in biomedicine are actually reproducible. This is highly suggested that many if not most published studies are actually false.

3) There are several problems in academia research. Downstream it's costing billions of dollars in private research and the lives of those who fall ill.

4) Academia should be formally teaching students how to conduct quality, repeatable  research and science. In biomedicine there is no formal course on this. In other words, students are conducting research as their peers do. Which makes them assume they are doing things correctly.

5) In biomedicine, repeatability and consistency issues arise with financial incentives, cell line accuracy and cultivation techniques, lack of animal testing diversity and control, statistical analysis, unintentional personal bias, method transparency, and general experimental design.

6) Producing quality science may demand more resources (time, money, animals, people, etc) but poor design is costing us far more in the long run.  The private sector invests billions running trials based on research where their mechanisms are often false and this ultimately is why we have made little progress in combating the major diseases like cancer, ALS, and heart disease. 

7) When presented with published articles I will ask some simple yet powerful questions to determine the validity of the study: 

Were the experiments performed blinded?  Were basic experiments repeated? Were all the results presented, or were some cherry-picked? Were there positive and negative controls? (A parallel experiment meant to fail) Did they use valid ingredients or cells lines that have been verified? And Were statistical tests appropriate? Or were they at least highly suggestive vs the standard "statistical significance" of 0.05%?    

8) SOPs are needed. One of the major challenges remains in the fields ability to enforce them.

9) The culture in biomedical science is in need of change. One where scientists are rewarded for rigor and publishing success as well as failures.

10) Finally, preventive medicine is still the best option! Exercising, nourishment, stress reduction, and lifestyle choices will likely continue to be the best defense against major disease within our lifetime.

(edited)
Comment URL copied!
Felix Fecher
May 8th, 2021 at 7:31 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis gives the reader an in depth look at the science and business that is connected to all kinds of medication. It includes everything: from not making improvements any in the last 40 years to the inaccuracy of scientific journals and the overall lack of quality in scientific research.


What I really liked about the book is that it made the issues within the field of biochemistry very obvious to the reader. You don't have to be balls deep in biochemical research to understand most parts of the book well enough.

 

The thing that got me most was the actual lack of progress in the last decades. For the average person like me, it is very hard to understand that there can be so much technological progress on one side, and such a big gap when it comes to really important things like medication.


It's really miserable, and the author does a a very good job of bringing it all to light, showing that the missing progress was mostly correlated to bad experimentation. For me, it was scary to read how many experiments weren't able to be replicated.


I see this book as an eye opener. The pharmaceutical industry needs to chase real science, not the next funding. If they would have not wasted the last 40 years of research, maybe some of the world's most dangerous chronic disesases like alzheimers would have probably already been cured.

Comment URL copied!
Alexander Mercieca
April 10th, 2021 at 1:48 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

The book, Rigor Mortis by Richard Harris, takes an in-depth look at the medical academic community as well as the urgency for the publication of articles for people wishing to either secure a job or advance in their field of study. I believe the author did a good job in pointing out many of the problems facing the industry such as the lack of reproducibility of the studies and the wide spread of misinformation based on flawed studies. I was surprised at the lack of accountability in the industries over the publication of bad information and the unwillingness for researches to admit when they made a mistake,

              The book opens with a story about how C. Glenn Begley decided to expose the amount of fallacious scientific studies published in the medical community. Not being in the medical community I was unaware to the extent of published journal articles which could not be replicated of were willed towards a certain outcome due to desire for researchers to have more funding by certain companies. I was shocked to find out that “one of the studies he couldn’t reproduce has been cited more than 2,000 times by other researchers” (p. 13). I feel that this would be a good starting point to use with clients in the gym and use that connection between bad publications to the bad diets out there.

              The section titled, “It’s Hard Even on the Good Days” painted a picture of how unsuccessful many scientific experiments truly are, but reiterates the idea “are we sure that the success rate of 11% of landmark papers isn’t a bonanza?” (p.39). I felt this was a good way to shift the mindset towards a more positive focus. The author then goes to point out how many of the experiments could have slight variations which cause much different outcomes, such as washing a test tube with soap and water versus acid. This would be a good analogy to use with clients in the gym to show them how even if they follow a rigorous workout regimen, if they do not have the proper nutrition then they might not get the same results as someone else following the same workout prescription. Its about the small variables.

              The next few sections that dealt with ALS and cancer spoke to me personally. My wife is currently going through treatment for stage 3 breast cancer and reading how a melanoma cancer cell was mistakenly use in research for breast cancer even though they respond differently to treatment made me raise questions to the number of women who have had reoccurring breast cancers since their doctors used studies based on the wrong type of cancer, so they received the wrong type of treatment. This section had me raising some questions.

              One quote I found very eye-opening was about how the mice reacted regarding stress and the results that had on the experiments. I found the statement on page 80 to be very thought provoking, that animals who are more anxious and stressed-out are more immune suppressed.  I would use this

with my clients at the gym to illustrate how effective using exercise as a way to destress could improve their health not only at the waistline but on a cellular level as well.

              The center of the reading really focused on making a very convincing argument for the lack on integrity in medical academic publications by presenting case after case where this was very evident. The concept of HARKing could be avoided if “they did a better job of keeping track of their ideas-especially if they documented what they were planning to do before they actually sat down to do it” (p. 146). This whole area of the book stood out because all I could think of was the fact that CrossFit has been an open-source way of collecting information on all the workouts and measuring the effectiveness of the program over the decades by gathering peoples scores on workouts and measuring progress. It is apparent that the way CrossFit documents their approach to health and wellness is far more reliable than even the “one-name journals”, as Harris put it. I would use this example with my clients as an example to the effectiveness of the CrossFit methodology and brand name.

              The book ends with some ideas about how the broken culture in medical publications can be corrected. The accountability aspect is one that I found to be very interesting. Using the examples mentioned in the text I would be able to show ways in which the FDA has made claims about overall health with the food pyramid, or give clients examples of foods that are marketed as healthy when they are actually just cleverly disguised hyper-palatable edible food-like substances or loaded with sugars.

              Overall, my takeaway from this book was that in all professions there is a need to get results fast. Those employed in the biomedical industry are tying to get as many publications attached to their name so that they can have upward mobility in the field, even though the people looking at their publication numbers most likely haven't read any of their publications. Speeding up the research to get results often produces bad results. This can also be true for a health and wellness program. Health and wellness is a life long journey and should not be marketed as a quick fix, rather a lifestyle change.

Comment URL copied!
Mary Lathrop
April 9th, 2021 at 6:35 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Initially I was shocked to hear about how much irreproducibility there has been in biomedical research and how much it has set us back from being able to make significant progress in our discoveries but the more I read, the more it made sense. It's not easy to reproduce an experiment EXACTLY the same, down to the gender of people conducting the experiment and what is even present in the room, one time, never mind multiple times, to make sure you continuously get the same outcomes. The section where Harris talked about the mice and how they respond differently to men versus women and how if a man's shirt is even in the room, they can smell it and will respond differently, was so intriguing to me! That's not something I would even think to consider when conducting an experiment so I totally get that in most cases the scenarios probably weren't as exact as they probably should be.


This definitely makes me consider tests that we do in CrossFit and how we use benchmark workouts. Technically, the majority of the time, we probably aren't repeating our tests EXACTLY as we did the first time. Should we be trying to be THAT exact? I know I don't measure the distance my barbell is away from the pull-up bar, have the same height pull-up bar, the same barbell, the same plates, and the same collars, etc. that I did the previous time I did the workout. Heck, I might not even be at the same gym! Does that make my PR invalid? I would say no, but my Fran time also isn't trying to cure cancer, so maybe it's just a bit less serious and it's ok to just repeat it as best as we can.


This book, however, did make me want to question everything I hear and read. This proves that there is always some tainted data out there and we need to look into things rather than just believing everything we hear or read. I often catch myself doing this, especially with the current pandemic. Everyone just spits out things they've heard without actually fact checking and it just becomes a giant game of Telephone. In the biomedical field, we see that careers and reputations are on the line based on simply getting something published. They get so caught up in the hype, the deadlines, and maintaining funding, that little things start to slide, sometimes big things, and sometimes complete hypotheses just to be able to say that something was successful or so it appears to be something ground-breaking. Therefore, the moral of the story is question everything.

Comment URL copied!
Alexander Mercieca
April 9th, 2021 at 4:40 pm
Commented on: 191128

The book, Rigor Mortis by Richard Harris, takes an in-depth look at the medical academic community as well as the urgency for the publication of articles for people wishing to either secure a job or advance in their field of study. I believe the author did a good job in pointing out many of the problems facing the industry such as the lack of reproducibility of the studies and the wide spread of misinformation based on flawed studies. I was surprised at the lack of accountability in the industries over the publication of bad information and the unwillingness for researches to admit when they made a mistake,

              The book opens with a story about how C. Glenn Begley decided to expose the amount of fallacious scientific studies published in the medical community. Not being in the medical community I was unaware to the extent of published journal articles which could not be replicated of were willed towards a certain outcome due to desire for researchers to have more funding by certain companies. I was shocked to find out that “one of the studies he couldn’t reproduce has been cited more than 2,000 times by other researchers” (p. 13). I feel that this would be a good starting point to use with clients in the gym and use that connection between bad publications to the bad diets out there.

              The section titled, “It’s Hard Even on the Good Days” painted a picture of how unsuccessful many scientific experiments truly are, but reiterates the idea “are we sure that the success rate of 11% of landmark papers isn’t a bonanza?” (p.39). I felt this was a good way to shift the mindset towards a more positive focus. The author then goes to point out how many of the experiments could have slight variations which cause much different outcomes, such as washing a test tube with soap and water versus acid. This would be a good analogy to use with clients in the gym to show them how even if they follow a rigorous workout regimen, if they do not have the proper nutrition then they might not get the same results as someone else following the same workout prescription. Its about the small variables.

              The next few sections that dealt with ALS and cancer spoke to me personally. My wife is currently going through treatment for stage 3 breast cancer and reading how a melanoma cancer cell was mistakenly use in research for breast cancer even though they respond differently to treatment made me raise questions to the number of women who have had reoccurring breast cancers since their doctors used studies based on the wrong type of cancer, so they received the wrong type of treatment. This section had me raising some questions.

              One quote I found very eye-opening was about how the mice reacted regarding stress and the results that had on the experiments. I found the statement on page 80 to be very thought provoking, that animals who are more anxious and stressed-out are more immune suppressed.  I would use this

with my clients at the gym to illustrate how effective using exercise as a way to destress could improve their health not only at the waistline but on a cellular level as well.

              The center of the reading really focused on making a very convincing argument for the lack on integrity in medical academic publications by presenting case after case where this was very evident. The concept of HARKing could be avoided if “they did a better job of keeping track of their ideas-especially if they documented what they were planning to do before they actually sat down to do it” (p. 146). This whole area of the book stood out because all I could think of was the fact that CrossFit has been an open-source way of collecting information on all the workouts and measuring the effectiveness of the program over the decades by gathering peoples scores on workouts and measuring progress. It is apparent that the way CrossFit documents their approach to health and wellness is far more reliable than even the “one-name journals”, as Harris put it. I would use this example with my clients as an example to the effectiveness of the CrossFit methodology and brand name.

              The book ends with some ideas about how the broken culture in medical publications can be corrected. The accountability aspect is one that I found to be very interesting. Using the examples mentioned in the text I would be able to show ways in which the FDA has made claims about overall health with the food pyramid, or give clients examples of foods that are marketed as healthy when they are actually just cleverly disguised hyper-palatable edible food-like substances or loaded with sugars.

              Overall, my takeaway from this book was that in all professions there is a need to get results fast. Those employed in the biomedical industry are tying to get as many publications attached to their name so that they can have upward mobility in the field, even though the people looking at their publication numbers most likely haven't read any of their publications. Speeding up the research to get results often produces bad results. This can also be true for a health and wellness program. Health and wellness is a life long journey and should not be marketed as a quick fix, rather a lifestyle change.

Comment URL copied!
Simon Mutch
April 8th, 2021 at 12:42 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

A lack of funding, pressure to produce results and scientist’s individual ego has led to research that have no reproducibility. It’s clear that studies which have reproducibility create solid scientific results that can be used to conduct new research. The scientific community should be chasing this standard over trying to get the result they want, as our own self-interests will always interfere with creating reliable research.

As a coach we know that creating a new cue for correcting loss of lumbar curve in the air squat may not give the desired outcome that we hoped for. However, knowing that it didn’t work repeatedly with the same athlete or others, allows us to find better ways of correcting that fault and finding one that repeatedly has the desired effect. We know the cues given to us by CrossFit have been tried and tested and are reliable in reproducing the desired outcome.

The pursuit of excellence reminds us that coaching is a work in progress just like scientific research. Failing is an important lesson that improves my knowledge and pushes me to become a better and more efficient coach, so I can create a positive and lasting change to my community of athletes.

Comment URL copied!
Nicolas Delay
March 17th, 2021 at 2:56 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

What I get from this book is that the scientific research model is deeply flawed for 4 main reasons :

► Economic first, mostly due to the system of fundings that modulates the incentives and the lab's poilitics

► Social then, due to our very foundamental need as human being to be recognized by others, sometimes at all cost

► Moral also, because modifiying a little parameter in an experiment to make a big change in career only relies on the integrity of the scientist, not the system

► Finally technical, just because we must admit that a great scientist needs to possess an advance level in multiple areas (statistics, human & animal biology, research metodology...)


Because this book mainly presents descriptive and documented facts, it's hard to refute it if you do not go deep into the litterature of the research world critics, which I honestly didn't do.


First, what M. Harris describes mainly revolves around the world of foundamental research, talking about cells, antibodies and only after about rats & mice. If I get to compare those statements with my experience of working with researchers in human sport physiology at the Sport University of Lyon, France, I must admit that the people i've been working with could not be put into this category, neither from a moral nor from a social perspective. They were working hard, carefully and with great passion for Science metodology from down to dusk. There is not a single event I can recall where I would I even doubt their honesty at work. Note, however, that I've no competencies to judge them in the two other points I listed above, first because they were my teachers, and second because I've never been exposed to a situation where my actions were fundings/money dependant.


How then, do I feel about that ? Well, truth is I see no paradox in M. Harris statements and my own experience in a researcher's team.


Again, what M. Harris describes mainly focuses on foundamental research : the luck with metformin, the forced validation of statins, and the golden subject of them all, the cancer cure. As stated, this field is as stressful as it is rewarding from an economic perspective, thus implicating the drift we've been talking about.


On the opposite side, Sport physiology wasn't under the spotlight of research at all. The lab I worked in was the smallest of the campus, and not very prestigious when viewed from outside. To me, because the market didn't really care about all this, the lab's politics and the researchers inside did an amazingly great job at working in line with their values and the research metodology.


All in all, this book made me discover the other side of the scientific research I didn't know. It will help me better explain some of the clear dysfonctions the world of human medecine is built on today, and the place we have as CrossFit trainers, to bring our elegant solution to the people in need.


Comment URL copied!
Alexander Mercieca
April 9th, 2021 at 4:51 pm

I agree with you. I was unaware how much funding was reliant upon the publications of the scientists at universities. I have been working in Education for 16 years and the sheer number of published papers with bad information being cites by thousands of research makes me being to question the validity of many of my own research projects I have done over the year which relied largely on published journal articles.

Comment URL copied!
Daniel Kachtik
March 14th, 2021 at 12:32 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Science is fallible. It is limited. It is a never ending work in progress. If experiment results differ in mice based on their environment how much more so do they differ in humans. If results differ on different tissue samples based on how and when they were harvested how much more so will the differ on humans. Science is driven by all kinds of motivations that have nothing to do with getting the best results. And even when that is the intention it is incredibly hard to get that result. The point I take from this is to trust our own empirical evidence. When I eat meat, vegetables, nut, and seeds, some fruit, little starch, and no sugar I am healthier and I don't get sick or fat. When I do constantly varied functional movements executed at high intensity I am healthier and I don't get sick or fat. I know this from myself for the last 14 years. There are times it is unavoidable that trusting in a pill is your best hope. This should be a last resort. It has never been more important to trust in what we can know by observing and that is there is a lifestyle that enables us to fight off the dangers of our modern world and this will work better than any man made cure.

Comment URL copied!
Gavin Heselton
March 9th, 2021 at 12:06 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

As a first book for the book club, Rigor Mortis issues a challenge to everyone participating: challenge everything. CrossFit is unique in the wider fitness industry in that the application of fundamental scientific practice is encouraged at the coal face. In our attempts to validate our own practice through “observable, measurable and repeatable” tests we align ourselves with the scientific community more than the ‘glossy magazine and protein shake’ side of the fitness industry. Even here in the comments, Drake acknowledges our charter of “mechanics consistency and intensity” in the execution of movement, as having parallels within the laboratory. We are experimenters in our own right, testing our own hypotheses and those of the wider community in the pursuit of physical excellence.


In the context of the provision of healthcare at a national level we are the primary care providers and our practice can directly circumvent much of the need for further medical intervention in that the most common conditions of chronic disease can be headed off by diet and exercise alone. Therefore it is a requirement of our ongoing development that we are aware of what happens beyond our level and to what extent it is or isn’t reliable as being restorative of health.


Having completed two research degrees and having worked in various clinical settings I, like Julie, was not surprised by the information presented here in the book. Obfuscation of data by bias, file drawer effects, conflicts of interest and the ‘standing on the shoulders of giants’ are first year statistics/experimental design issues. Whilst it is unsurprising that they persist into professional practice; it is nonetheless frustrating to see that regulation of these industries is insufficient to prevent them. What is more disturbing, is that poor scientific rigour or worse; deliberate falsification of data is in many cases actually (directly or indirectly) incentivised for and by organisations or individuals.


The narrative of influences (primarily financial) external to the laboratory affecting experimental outcomes is supported in other books within the field. Mike Osterholm’s book ‘Deadliest Enemy’ ominously foretells of our current predicament with COVID-19 – there is little to no government funding for ongoing vaccine research as a proactive defence against global pandemic. Private industry carries the burden of this expensive R&D and where companies have previously developed vaccines for apparently imminent global pandemics (SARS-2, MERS, Ebola) out of their own pockets only for the disease to be contained by non-pharmaceutical intervention; it is apparent from a fiscal standpoint there is no incentive for them to be proactive participants in this process. By making this work everyone’s responsibility it is, in fact, nobody’s.


I do not profess to have an answer to reducing these fundamental issues of bias and poor experimental design in practice, nor do I know how resource-hungry research and development within bioscience can be unshackled from the external influence of finance. What I do know however, is that by bringing to light that these practices persist in our scientific community, we can begin to hold ourselves and our own peers to greater account and therefore begin the arduous task of a global paradigm shift to quality of practice, over quantity of profit.

Comment URL copied!
Alexander Mercieca
April 9th, 2021 at 7:01 pm

That was the one takeaway I got from this reading as well! I gained even more of an appreciation of the transparency that CrossFit provides through the open source format of their programming. I can only imagine the difference in the medical community if they provided the same amount of transparency in their studies.

Comment URL copied!
Karen Katzenbach
February 22nd, 2021 at 2:21 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

In Rigor Mortis provides a glimpse into the highly competitive world of biomedical research. A world where publication in prestigious journals leads to advancement and funding from commercial interests has more sway than than the scientific rigor that author Richard Harris takes aim at with the book's title.


It's not surprising that bad science slips through the cracks in the system. After all there are hacks and those willing to take shortcuts in any profession. The surprising and concerning aspect is that the bad science does not get called out even when it is identified. In fact, once published, it's likely to be cited often in other studies so that it persists as truth even if a retraction has been published, which is rare.


The effect on the marketplace has been profound. Cures and treatments for medical conditions have proliferated, bolstered by studies that are not reproducible and yet not called into the question. Time-starved doctors are not able to do their own research and the consumer is left at the mercy of a system that does not have their best interests at heart.


Harris brings all the issues behind the decline of science rigor to light with compelling and at times heart-breaking stories. This, no doubt, is the first step in changing the broken system but it will take a much larger moment to fuel true change. Informed consumers of biomedical research, from the physicians employing the cures to the patients receiving them is the only way to make true change. For that to happen there must be more transparency around the methods used during research and more accountability for those who skirt the system.


Will this happen soon? Not likely but with a growing number of health & fitness professionals outside the system becoming more informed, I'm optimistic that a movement in the right direction is starting.

Comment URL copied!
Ryno Verster
February 20th, 2021 at 2:05 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I do completely agree with Harris’s view that  fundamental issues exists in the science research studie arena that could lead one to think, is any scientific study reliable?


Then on the other hand the trail and error of science research studies managed to extended the human lifespan materially compared to when no medicine was available.


Harris estimates that about half of the U.S. government’s $30 billion annual investment in biomedical research yields results that are seriously flawed and can’t be trusted.


With that said his book is concerned with more with laboratory-based, preclinical, biomedical science, rather than clinical research that involves human subjects.


Here is my opinion about science and its importance in society:


Human nature is the number one factor that allows research to be untrustworthy. A study could easily be manipulated to: favor personal biased results, or a good intended study could involve human error, or alternative motives for example to gain financial advantage could be the drive.


The problem here is that new research takes previously documented and accepted studies as a base and even if the research is good intended, it may be based on a previously flawed study that involves error.


With that said, should we trust scientific research at all? Is there a place for it?


Again in my opinion after listening to this audiobook, it is clear that the advances in medicine that saves lives on a daily basis clearly outweighs the negative aspects of the research.


I support scientific research and just as there is injustice and corruption in all aspects of life, there is also good and trust.


I for one believe in the good side of humans and that the truth will always triumph. Therefore scientific research must continue.

Comment URL copied!
Dennis Sukholutsky
January 28th, 2021 at 3:09 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This book tackles one of the reasons modern medicine has made minimal improvements in treating disease, cancer, and other maladies. This reason is poor biomedical science and studies.


Medical companies look to medical studies for new drugs to develop. Unfortunately, many medical studies are not reproducible. There is unconscious bias among scientists when it comes to selecting populations and excluding results. Measurements can be incorrect during medical studies.


One reason for this is due to low scientific funding availability. This leads to cutting corners during studies due to high medication costs. Harris presents an example where a large ALS study was invalidated due to cutting corners because of high medication costs. This led to a review before the Senate and now the NIH requires better study plans to receive grants.


Another problem is that some medicine that works in mice does not work in humans, and vice versa. Minutia like the music, temperature, and handler of the mice can lead to different results between labs.


There are also problems with cell classifications in cell labs. Computing power allows testing on millions of cells, but the cells may be old or misclassified. Scientists can also make errors with population selection, causing batch effect issues. Sometimes exploratory studies end up being passed off as confirmatory studies so the study can appear successful.


Labs compete with each other for breakthroughs and publications, so they are hesitant to share data. Scientists scramble to publish first in science journals, leading to errors and retractions.


The takeaway is that scientists need to collaborate and slow down due to the unsustainable pace and frequent errors. As a result, science will be better and more effective, hopefully speeding up medical discoveries.

Comment URL copied!
David Hallmann
December 6th, 2020 at 5:37 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It all begins with the statement: Half of clinical studies aren´t trustworthy and only 15% of studies are correct.

In the further chapters the reader get to know why...

Be it because tissue is simply not the tissue they supposed to use for a distinct study (like mentioned in the breast cancer study) or simply because mice aren´t human.


All the cases in the book are a really good example to ask questions for yourself and find out things by experience. It is not seldom that in one study one thing is healthy and good but in another one it´s unhealthy and not good. How often do those things change over the years?!


The lack of a uniformly and international way to proceed studies mixed with a state of lack of time, lack of money and surely not a lack of lobbyists is scary.

Science has become a "hyper-competitive rat race" and that´s actually the main problem. They should be driven by curiosity with enough time to do proper experiments and science in general.


"Today the pressure puts the best scientists in dangerous territory."

Due to this pressure, the book mentioned, they spend to less time in statistics, study design and other fundamental things.

When those scientists run into a problem they do other experiments and don´t do a lot of reflection about the things that are going on. And that is why one of the solution can be to bring philosophy back into biomedical science.

I like that idea which is mentioned in chapter 7. Because as an affiliate owner in these Covid-19 times I absolutely can relate. I´m by far not an scientist but I have to try things out. If I don´t take time to reflect what I did and do from time to time I can´t know if those things are working.


"It pays to be first." "It doesn´t necessary pay out to be right" That leads to sloppy science and to cut corners to be first. And just because it keeps the money flowing in it has to be quick and that doesn´t lead to bold and deep insight.


So why not only teach future scientist how to think like an scientist but to give them a "how to..."?

How to design experiments before teaching facts which would give a scientist more room to do their experiments with a free mind but with a method. A free mind doesn´t hang stubborn on ideas that doesn´t work.

It also would lead to more bravery amongst scientists because they would be more brave to destroy dogmas around the biomedical science. That´s a way to move forward.


In the book I also found out that there are ways to reduce those failures in biomedical science. Just by open up and use the technique we have.

A good example is an increased transparency could lead to a reduction of the problem of reproduction because just by publishing the hypothesis in advance.


What I learned from this book is to go my way, to be brave and to trust my own experience and "science".

To keep things simple and try new things but also reflect these from time to time and in stillness.

To escape from the rat race to think about my ideas as an affiliate owner and coach. So this book is not only about scientist... We all live in a world where everything has to be quick and immediately done. But that´s not always how things work. Especially if they should be high qualitative.

Comment URL copied!
Case Belcher
November 29th, 2020 at 6:44 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

“Rigor Mortis” opened my eyes to the vast amount of faulty medical research out there.


I came into the book with a mindset that I believe is not unlike many others - Having the idea that medical science had progressed to an era where most studies are accurate and reliable, thus producing safe and effective treatments.


Harris (the author), who’s covered science and medicine for NPR for decades. Makes a clear case uncovering the data and outlining the pitfalls in biomedical research. These pitfalls lead to billions of dollars wasted each year, stalls in the advancement of science, and unreliable cures.


The book illuminates the fact that medical research is a marketplace ran by humans - both of which have natural and inherent flaws. 


First, medical science is costly, slow, and strife with moral considerations, so bringing a cure from a hypothesis to the market is inherently hard with many steps along the way, and each step creates its own opportunity for error.   


Add to this that humans make errors, machines make errors, too many scientists are chasing too little funding, there’s lack of standards and methods across studies, lots of profit depends on the outcome of studies, and scientists are motivated by a culture of career advancement… You begin to see why much of medical science lacks rigor and reproducibility. 


Some cogent examples from the book that stood out (1) Likely only 15% of biomedical studies are accurate. (2) Only a similar percentage of studies are double-blind. (3) We rely too heavily on animal models. (4) A good percentage of current studies site data from older inaccurate or retracted studies.


The shear amount of possibility for error uncovered in biomedical research is astounding, so I don’t fault the scientists, but I do believe the systems and marketplace surrounding biomedical research is in need of improvement. This is apparent from Harris’s examples of the number of organizations sprouting up, along with the field of meta-research, whose soul purpose is to assess the validity of existing studies and to create better standards and practices in the field. 


In my opinion, the funding of biomedical research is the linchpin in creating more accurate science, but I don’t know if the solution is better policies with our current government funding model or through more private funding. 

 

I don’t think we should lose all faith in medicine, but I do believe as consumers we should approach medical interventions with more skepticism and thoroughness. This book (and a few others) have presented enough evidence for me to read beyond the headlines. I now look to see how a study was conducted, who supported or paid for a study, and I look for other supporting studies or a larger body of evidence. 


It also drives home the importance of prevention. With so many opportunities for error in medical interventions, a healthy diet and exercise seems like a much safer bet. 

Comment URL copied!
Oscar Isacsson
November 24th, 2020 at 6:28 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis - Bookclub


For many of my clients, I lean on studies to help make recommendations. This has given me some guidance on what to recommend and I believe it helps the client to buy into the process. It’s discouraging to see how low the success rate for reproducing a study truly is. This doesn't sway me from looking or recommending research but it does make me read their conclusion with a grain of salt. 


There are many factoids that stand out to me in this book. 


  • 30 Billion dollars of tax dollars spent on research that can be very questionable
  • The reliance on very few mice to extrapolate results
  • Only 60% of results can be predicted from mice to other rodents yet human is a logical comparison


You often see researchers referencing other studies and building off their already established premise. So many layers on top of layers that are supposed to lead to even further advancement but could be very hard to reproduce in the first place.


During my undergrad and graduate degree, I spent a good amount of time in the exercise science lab. I got familiar with the protocols and equipment for the more common exercise testing procedures. VO2 max testing, RMR metabolic testing, blood lactate levels, BOD pods, 1RM testing, and etc. A few months ago I returned to my old school to partake in a study. The students running me through the testing were unaware that I was familiar with the protocols. What I experienced were well-meaning students that had been taught by other students. Small differences in calibration, execution, language, and etc. might not seem like a big deal but it really adds up over the whole battery of tests. 


The VO2max is absolutely brutal and I’ve scored in the high 50s several times. For this most recent testing, I scored in the mid-20s which I know is incorrect. The grad students however dutifully recorded the value and moved on. 


This is a huge problem with studies. There are so many tiny factors to account for and make standard across the board. Rigor Mortis showed me that you can’t take published science to be 100% true. Going forward I will continue to read the research and recommend studies to other people as a form of spreading information and creating buy-in. I will however be more apprehensive towards adopting a paper as fact and emphasize to my clients that they need to be doing their own experiments. 


How do I feel after 8 hours of sleep versus 5? What if I cut out soda? Do I feel better with an extra rest day or not? 


Trainers and clients ultimately need to keep track of their own data as closely as possible. Limit the number of variables they change at once, and steadily work towards increasing work capacity across broad time and modal domains. 

Comment URL copied!
Stephanie Smith
October 9th, 2020 at 11:25 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

“Rigor Mortis” is a thorough examination and critique of the biomedical research industry and the systemic pitfalls that are hamstringing scientific progress. It is—as many others here have succinctly identified—disturbing. From poor design study to tainted test materials, nearly everything a researcher touches can be, and often is, affected by bias, design flaw or material contamination. Every chapter is a new trip down another rabbit hole of potential ways scientific research can go awry.


Chapter 3 specifically discusses how poor experimental design and the resulting irreproducibility of studies lead to failed trials and are thereby massive, unnecessary drains on funding. Perhaps even more concerning is that human lives are caught up in these flawed efforts, with hopeful, disease-afflicted patients volunteering to enroll in drug trials that are just destined to fail.


This was of particular personal interest because two immediate family members have recently been affected by pancreatic cancer and Duchenne muscular dystrophy; these are devastating diagnoses that have a fast-track to one outcome—death. Because of the bleak prognoses, drug trials often become the singular beacon of hope for patients. Families travel the country, spending thousands of dollars to get our loved ones screened for and enrolled in a trial in an attempt to prolong their lives. There is an inherent trust and belief that the doctors and researchers involved are confident in the trial’s science and see promise in what they are testing. It’s striking to see the proliferation of avoidable failures when your own family members lives are dependent on the science being right.


We see this sentiment reflected in the author’s retelling of Tom Murphy’s ALS diagnosis (p. 53), where we are told that Murphy enrolled in a “promising trial” that was ultimately a bust. We’re then told that the original ALS animal studies were fatally flawed because of faulty study design, and upon restudy, not a single developed drug showed signs of promise on ALS afflicted mice. Even after the foundational science was ironed out, funding became an issue. This resulted in a number of drugs going to human trial without “first making sure the scientific underpinnings [of the research] were sound” (p. 57). The result? Failures with each of the drugs.


In each chapter we see additional ways in which time, money, and as in the above cases, human lives, are essentially wasted due to the titular problem of “sloppy science.” Harris does leave plenty of room for optimism, though—he makes clear in the same chapter that there are factions of scientists, government officials and philanthropists alike who are on a mission to develop and instate “consensus standards” for biomedical research. In regard to neuromuscular diseases specifically, Harris details the efforts of two advocacy groups—Cure Duchenne and Parent Project Muscular Dystrophy—which require studies be thoroughly vetted before they will agree to funding them. Harris also discusses TACT, a now self-funded committee that is a “no-nonsense venue for reviewing potential drugs for neuromuscular diseases” (p. 65-67). These types of standards and review processes will help ensure that studies destined to fail for “stupid reasons” are more likely to be weeded out before too much time, money or human life is invested.


This is, thankfully, not the only area of potential redemption for scientific research in the book. Harris does a great job of highlighting what scientists are doing to expose and correct the faults he details in each chapter, with nearly the entirety of Chapter 10 spent discussing a path “out of the structural morass” (p. 236) that biomedical research currently finds itself in. Whether it’s better educating the next generation of researchers in scientific procedure, restructuring how post-doctoral researchers are used, establishing university endowments for tenured professors, creating revenue streams that incentivize clean academic research, or improving the peer review process for journal publication—there are myriad ways biomedical research can re-center itself and find a better path toward scientific advancement in the 21st century. 

Comment URL copied!
John Singleton
June 22nd, 2020 at 7:54 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis, this was my first time coming across the book. The title is certainly strong and clearly defines what is laid out in the pages.


“How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions”


It was sad to read that the scientific research currently happening in biomedicine laboratories, is failing or being stalled because of a variety of pressures and that shortcuts are being taken by scientists in order to race to submit papers to the main scientific magazines.


For someone outside of the scientific community you immediately “believe” the research, because we have an innate trust. This book did made me question this trust.


However it did seem these rushed reports are not primarily because people are intentionally messing up, it seemed that our nature is what causes many problems; such as seeing what you want to see, and not seeing what is actually there.


This leads the conclusion that we must do better, we must hold ourselves to a higher standard and follow standardised procedures, it will take longer, however the results produced will be invaluable in actually moving medicine forward.


It was also interesting to see Julie Fouchers comments regarding the book, unfortunately confirming what is stated even within a clinical setting. Having someone so prominent in the community confirm this made it seem so much more real.


(Listened on Audible)

Comment URL copied!
Thomas Grotz
April 24th, 2020 at 9:14 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

In “Rigor Mortis” (2017), Richard Harris’ first nine chapters serve as a thorough evaluation of the current practices of modern biomedical research and their flaws, with the tenth chapter offering tangible solutions to better the industry as a whole. The book’s title is not only engaging but also defines the dire need for scientific studies to change. ‘Rigor’ means “testing whether individual studies can be reproduced” (Harris 22), yet many of these major studies, from ALS, to animal testing, to misidentified cell lines/stem cell research, and to ovarian cancer, lack thorough and sound testing due to “perverse incentives” (Harris 78) such as funding (grants), title (power), recognition (influence), and timing (who will publish first).


The “hyper competitive rat race” (Harris 106) that the author describes as the biomedical research industry reminds me of how today’s fitness industry can also be, with its professionals striving to find ‘the best’ methodology, workouts, or protein powder. However, CrossFit’s measurable, observable, and repeatable data, the "relentless constant pursuit of excellence" (Glassman), and its emphasis on virtuosity (doing the common uncommonly well) in fitness and nutrition align with Harris’ intended approach for biomedical research to improve. As a Certified CrossFit Trainer, part of my scope of practice involves designing workouts for both groups and individuals yet also recognizing the limitations of my own knowledge and thus referring my clients to other healthcare professionals when appropriate. The importance of thorough research and ‘reproducibility’ is inherent in both CrossFit’s strength-and-conditioning program and sound biomedical research.


Solutions to the faultiness in today’s biomedical research can be remedied by “getting individual scientists to change their ways, getting journals to change their incentives, getting funding agencies to promote better practices, and… getting universities to grapple with these issues” (Harris 139). Given the current COVID-19 crisis, I imagine there are many “perverse incentives” driving biomedical scientists. Timing is of the essence, which may in fact affect the meticulousness in findings and thus the effectiveness of vaccinations.

(edited)
Comment URL copied!
Tom Henry
March 23rd, 2020 at 1:27 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis was an excellent recommendation to start off this book club. Until you read a book like this (or you're actively involved in surveying current literature about scientific studies), it's hard to know just how deep the rabbit hole goes. Although I have been involved in the health and fitness industry for a decade, in the last few years I have been more confused than ever in regards to what is regarded as "the truth" when it comes to health. This could be down to the open forum nature of the internet but more so, what's being pumped down to us via "reliable" government recommendations. Little did I know just how corrupt this hierarchy is and how difficult it is for even health professionals to interpret this information and recommend healthy lifestyle advice to their patients and clients, let alone the average Joe that is trying to keep themselves healthy without access to a personal trainer or doctor. In the second part of this book, I was even more surprised to hear that this corruption and misinformation is not limited to our diet but also in the advice and disclosure of information around serious life threatening diseases such as cancer. With many people misplacing their trust in the system, agreeing to long and arduous treatment programs which have been scientifically proven NOT to be effective, are nevertheless still being push out to the population as standard operating procedure. Last year, my aunty passed away from breast cancer, my family were telling me how hopeful they were after speaking to her Doctors but now after reading this book I am aware of how low her chances of survival were if she was to continue following the prescribed treatment plan.

I recommend this book to anyone who is involved in the health sector but also anyone who has a loved one or themselves facing a long term treatment program for a health condition or disease, as it may give you the confidence to ask the physician "WHY?"

Comment URL copied!
Brian Chontosh
December 14th, 2019 at 4:32 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Finally getting through everyone’s comments after listening to the audiobook. Much of what has already written rings in tune with my thinking. I especially thought Drake Sladky had a pretty interesting point about “M-C-I” and Matt Lodin churched it up with Morals, Courage, and Intensity. That made me think a little deeper about my point of significance from the book; Self Bias (or any number of the other types of biases out there - selection, conclusion…)


I’m looking at it from a point of not being ignorant to the finer aspects of the concept and even actively try to defuse it from personal activities and thoughts. No matter how hard you try it is always present. Tackling it from a singular or buddy pair or a group doesn’t really disallow the concept. It certainly protects in some regard, but it only evolves. There’re dozens of eponymous laws out there that make plenty of light to this through their wisdom (observation & ingenuity?).


I think attempting to predicate intent and not inferring motivation is required to temper reactions from this book towards any individual or group of scientists is important. Mostly reserving dismay towards collective practices and not being surprised by the corruptive forces from specific interests and human nature.


So, at large, I don’t think it’s quite a function of M-C-I at the origin; by either Drake or Matt’s pronouncement; although certainly a byproduct.


Looking at it for myself, as I dork in the field of leadership vs health practice, I’m looking to further buffer from how bias affects my personal narrative and self validates my own bullshit, comfort, weaknesses. Then to parlay any success here forward to creating awareness for others. But that requires being brutally critical, frank, honest, and sterile. And we all know most do not enjoy receiving this let alone providing it to others, i.e., enter feelings. Sure, tact can come to play, but really, that just opens the door for misinterpretation - deliberate or not.


Whether Rigor Mortis as a first selection for the book club is fortuitous or by design from an aspect of the critical & frank conversation of an industry to do some hard self assessment, correction, and stabilization of bias affects/effects matters little. For me, that is my takeaway - that this is significant part of the intent CrossFit stakes towards the fitness and health industries.


For the record, I am not committed to believing that all bias is bad; however, it is required to be always assumed and acceptable/appropriate.

Comment URL copied!
Ginger Sladky
December 7th, 2019 at 11:01 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

When we don't question long-standing procedures, even after we witness their ineffectiveness. When we refuse to make waves, and lay still even when everyone around us is sinking. When we fail to shine spotlights and make our outrage public, in the face of blatant deception. We are part of the problem. CrossFit has never been afraid to call out corruption. And bullies. And incompetence. And inaccuracy. I hope everyone here uses their social media channels to express Outrage so that we can elevate awareness of the conversation. If my brother had been privy to the full set of raw data documenting any success & all the failures of the experimental treatment that killed him, would he have decided against the procedure? Transparency of all clinical trial data is a project Ben Goldacre is spearheading. He has a petition supporters can sign. http://www.alltrials.net/find-out-more/why-this-matters/

We must show Outrage to raise awareness of the importance of Transparency, and to push Accountability of everyone responsible for allowing sloppy or deceptive research to be published or marketed. Together our waves are bigger.

Comment URL copied!
Dorota Stenclova
December 5th, 2019 at 9:55 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It is great to read all the comments from all of you. A lot of thoughts about Rigor Mortis were already said.


For myself the book represents an alarming insight into what I thought that was going on in biomedical research, but I never realised how bad it really was. The health industry is a mess. This book is a wake up call and my only hope is that we can learn from it being more critical to research results and hopefully having more quality than sloppy science in the future.

Comment URL copied!
Alec Zirkenbach
December 2nd, 2019 at 11:03 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This book is a MUST READ for anyone in the fitness community!!!


WHY?: It'll give you the ability to critically look at all of the "noise" coming out of clinical studies. You'll be better prepared to understand what studies have credibility versus potential bias or faults.


WHAT YOU'LL LEARN: This book highlights the common bias in study funding, professional pressure to publish anything, faults in study design, poor or biased data interpretation, and so much more.


EXAMPLEMy most mind-blown chapter is titled "Misled by Mice." The author highlights the false assumption that a mouse's biomechanisms will be similar to a human's. Let that sink in for a minute. We study mice and think that if a potential drug or exercise protocol works for mice, that it should work for humans...but we (humans) are not mice! And we shouldn't expect our biology to react the same way. SMH. If this were true, there wound't be any need for veterinarians...they'd just be called Doctors of all animals. But since animals are different than humans, we need specialists (veterinarians) to care for the specific biologies of animals.


Did you know that over 10 Million mice are used in lab studies each year (that's just the USA). Nearly every study uses mice because 1) everyone else uses mice and 2) mice are considered a "model organism" used in basic biology studies and drug research.


And it gets worse...the book highlights how some studies were being re-tested for validity in rats instead of mice. Rats and mice...same same, right? So the results should be the same, right? Nope!!! Mice do a "so-so job of predicting what will work in a rat (~60% or less of the time)...we should be very humble about what they tell us about human beings."  Yikes!

Comment URL copied!
Embriette Hyde
December 2nd, 2019 at 10:52 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It has been difficult for me to figure out where to begin, because there is so much I could say about this amazing piece of work. As a PhD-trained woman in science, what I can say is this: I can count the number of times I've come up against gender bias in my career on a single hand. I need double that, if not more, to count the number of times I've personally experienced every single example laid out in Rigor Mortis, from under-powering experiments due to lack of funds to reporting significant p-values on experiments that were exploratory. I don't say this to minimize the importance of the gender equality effort, I say this to highlight the severity of a problem that I still can't believe exists in science today.


This book had me remembering my Graduate School and post-doc days in more ways than one, but there was one discussion that got me especially fired up. On page 164, the author is talking about the graduate school curriculum and how it is painfully lacking in teaching scientific methods. How woefully true that is. I remember my classes. Yes, they taught facts. No, they didn't teach how to design scientific experiments. Really, they were just accelerated versions of classes I took during my undergraduate years, and something to blow through as quickly as possible so I could get to the good stuff, what I went to graduate school to do: research. One positive thing I can say about my classes -- which was specific to my program, one of twelve at the College -- is we did learn how to read and dissect scientific papers, especially to determine whether the science was rigorously done or not. Unfortunately this was a skill that wasn't put into practice in the laboratory that I eventually joined. It wasn't until my post-doc that I even learned to write papers while asking myself "what would a reviewer say?" In reality, I shouldn't have been fearing the reviewer. I should have been trying to make sure my science was sound, whether it was a reviewer that pointed it out or someone else.


Another theme the surfaces time and again in Rigor Mortis is industry vs. academia. I quit academia two years ago to become a writer, battling hype and public misunderstanding and misinformation about science. My first job as a freelancer involved writing for an organization that brings together companies in the synthetic biology space. I was blown away by the differences I saw between industry and academia. I was certain, as an academic, that the science I was doing was pure and unadulterated by money, unlike industrial research which was certainly driven by financial incentives to be biased. I couldn't accept the fact that the majority of companies -- especially startups that are most certainly not driven by money -- were leading the charge in developing tools to design experiments and enable reproducibility, taking the time to repeat experiments and do everything necessary to be able to claim something about their product, and so forth. Why? Why is it this way?


Money (or the lack thereof in academia) is huge, and this is discussed at large in Rigor Mortis, too. It seems to be a never-ending, cyclical issue. We really need to do something about the current system. What that is, I don't know. But the future of science depends on it.


Kudos to CrossFit for highlighting such important works and fostering discussion. This is something everyone -- scientist and non-scientist alike -- needs to be talking about. It's encouraging to see a company with such a large, influential platform leading the charge.

Comment URL copied!
Jason MacDonald
December 2nd, 2019 at 9:23 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I'm not sure what I could write that hasn't already been said or observed here. Overall this is a really frustrating read, bringing to light that allot of what we read and are lead to believe is in fact bad science, sloppy work and personal agendas. Not surprising why coach Glassman is so determined to expose it for what it is. Even more frustrating is that in some situations this sloppy science is being used to prevent good people from doing great things that are really helping people. I'm proud to be a part of a community willing to fight for what is true and right, putting service to others ahead of self promotion.

Comment URL copied!
Lachlan McGonigle
December 2nd, 2019 at 8:46 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

After reading Rigor Mortis, I decided to do some research and talking to people I knew in the industry. After doing so, I learned that what was said, is mostly true. It seems that scientists in the industry have the same concerns as those judging them. As with most things, people are a product of the circumstances placed upon them. In order to receive a grant from the NIH, one must go through a rigorous process, and only 10-15 percent of applications receive a grant. Though, once the scientist receives the money, it seems that there is little to no follow up on whether what they are doing and have done is worthwhile. They are asked to publish in order to receive, but publishing papers doesn't seem to require the same rigor as that of receiving a grant.


As Joe Masley mentioned above, it seems that there is a distinct lack of uniform standards, if any, for conducting research. With pressure to publish, and a lack of guidance from a somewhat non-existent mentor program, maybe it is time for the NIH to begin the change in the system. The head of the NIH, Director Collins, has acknowledge these flaws in the system. It seems that the NIH needs to provide both a robust follow up, and or peer review program for publishing papers of all kinds along with a greater emphasis on the guidance of proper methods of practice through PHD programs.


As a collective, I think those of us with a vested interest in Biomedical science (all of us!) can not only continue or research into this industry, but continue to champion and support those doing good research!

Comment URL copied!
Patrick Barber
December 2nd, 2019 at 5:40 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It has been amazing reading through some of my favorite people's mini book reports and seeing what they have taken from Rigor Mortis! Thank you all for taking the time to share your thoughts on the book.

Comment URL copied!
Maggie Tincher
December 1st, 2019 at 3:55 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Great comments everyone!


Researches and studies, whether they are biomedicine, clinical or drug related, are meant to increase and deepen our knowledge. Unfortunately, because of the “publish or perish” culture, the competitiveness, and maybe embarrassment, this is not what we get.


Instead of collaboration, what we see is a field of professionals and academics that are afraid of sharing their findings/data/methods. If data-sharing is needed in order to ensure that those studies can be reproduced and replicated, why this is not happening? Why are the studies not being shared? Is it because of competitiveness? Is it because the results are being manipulated towards what the researches want to publish? Is it because academics don’t want to be embarrassed in front of their peers? Maybe all of the above.


Maybe the problem here is not just the research (or the individuals doing the research), but how the value and worth of their work is evaluated. What we see is a race for breakthroughs, and this race is causing patients to pay a price... with their lives. Because there is not enough funding and the pressure to publish high-impact findings is pretty high, what we see is scientists ignoring or fabricating data, prioritizing studies, cutting corners, and hyping results. This rush to be the first ones to publish a new research, even when the outcomes are inconclusive or don’t meet the expectations, is putting patients are risk. This “publish first, correct later” culture has to change.


So, is there hope? I honestly don’t know. Do we need another set of regulations? Maybe. Will these standards increase rigor in performing and interpreting experiments? I don’t know, but I really hope so.

Comment URL copied!
Benjamin Tausch
December 1st, 2019 at 12:48 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Great Book!

Comment URL copied!
Julie Foucher
November 30th, 2019 at 10:25 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This was a fascinating read and although I wish I could say a lot of this took my by surprise,

I’ve unfortunately been exposed to many of these grim realities of biomedical research through my own training. As I read through the book, I also couldn’t help but draw some parallels between the biomedical research field and similar issues facing clinical medicine.


As many others have discussed here, one of the major drivers of this mess is the misaligned incentives that exist in both academia and industry. A career as an academic researcher is not an easy one - jobs are scarce, and once you do land a good job, you have to work like crazy to obtain enough research funding and push out enough publications to keep it. There are so many other job options with lower stress and higher job security, I imagine you have to have a very strong calling to research to pursue this path. As discussed in the book, rather than being incentivized to ask good questions and conduct rigorous studies no matter how long they take, scientists are incentivized to obtain more grant money and publish more papers which inevitably leads to asking questions that can be answered easily and quickly and with few resources. As noted on p 225, “The push for quantity is utterly misplaced…scientists aren’t asking questions with important answers; they’re asking easily answerable questions. Nobody wants to risk spending four years on a risky research project with a big potential payoff, but that could also fall flat.” Ultimately the scientist’s career security is driving the research being done, not a pure desire to answer important questions that will move the field of medicine forward and ultimately help patients. It’s no better in industry-based biomedical research, where the scientist’s career is not on the line so much (if a project flops, they’ll most likely just switch to a different project within the company) but the company’s bottom line becomes the main driver. This leads to asking questions and studying therapies that will be likely to turn a profit or have a large market, not necessarily what is best to move a field forward. These pressures lead to unbelievably high rates of questionable behavior in scientific research; with studies cited on p 186-187 finding 1/3 of scientists admitting to questionable practices and 14% observing serious misconduct in research.


All of this begs the question, who is looking out for the patients? None of the incentives above have anything to do with doing what’s best for patients. As one example of this in research, on p 59, NIH director Francis Collins states that he “was stunned when he heard about the pointless ALS trails that taxpayers had funded. ‘Humans were being put at risk based on that kind of data, and that took my breath away.” Too often patients - who are the most vulnerable group in this story - get taken advantage of or receive subpar care. Improving patient care should be the ultimate driver of research, but sadly patients are often the victims of this system whose incentives are driven by careers and money.


I think this excerpt from p 163 sums up the issues created by this system well: “‘Why did you go into science in the first place? Didn’t you go into science because you wanted to make the world a better place?’ Yeah they did but that was when they were a grad student or an undergraduate. They’ve forgotten that long ago. They’re in the rat race now.’ They need to get their next grant, publish their next paper, and receive credit for everything they do.”


I find this to be true not only for scientists, but also commonly for clinicians. I think most people go into careers in medicine because they want to make the world and better place and help people, but I often see this enthusiasm and vigor wane over the course of one’s training such that by the time you are practicing as a researcher or a physician, you’ve been so beaten up by a system with perverse incentives that it’s hard to hold onto that intention for why you chose this path in the first place. We need to work to address the system issues to realign incentives in such a way that we can capitalize on our passions to work toward better solutions for the people who need them, not to stifle these passions in a so-called “rat race” just to make a living and to improve the bottom line of academic institutions or corporations.


As a result of these misaligned incentives, we’ve ended up with an explosion of low-quality information, and it’s hard to know what to trust. Studies that have later been proven to be invalid are still being cited and used to generate new hypotheses, further contributing to wasted time and resources as well as more noise to sort through in order to find good science. One of the studies Glen Begley couldn’t reproduce “has been cited more than 2,000 times by other researchers who have been building on or at least referring to it, without actually validating the underlying result.” (p 12) In another example, “Years after two of the largest and most expensive medical studies ever undertaken had debunked the claim that vitamin E reduces heart disease, half of all articles on the subject still cited the original study favorably” (p 219). This reminds me of a lot of the noise that’s been generated in our clinical record keeping systems since the rise of the EMR. So often now we see errors as a result of information being “copied forward” in a patient’s chart without being actually validated as accurate. Again, why do these things happen? In order to meet the pressures of publishing, research funding, insurance and billing - not because it’s the right thing to do or what’s best for the patient.


Both research and clinical care have also become increasingly siloed which often prevents creativity and the ability to understand the big picture. As noted on p 165, “scientists need to spend more time thinking broadly about science and less about the specifics of their discipline. That creates intellectual ruts, which, among other things, make scientists cling more stubbornly to ideas that could be wrong.” I would say the same goes for our increasingly sub-specialized clinical system, where we have incredibly bright people working on a single body system or disease process, but fewer and fewer people looking at the big picture. The problems with this approach become apparent when you think about the complexity of the human body and how little we truly understand about how it’s systems work together. We may choose one molecule to target with a drug, but “evolution has created so many redundant systems that targeting a single pathway in a complex network will rarely work. Diet drugs are a good example.” (p 89) Our bodies are complex, dynamic systems that need to be studied as such, not as isolated parts that don’t talk to one another.


As a clinician, this mess is extremely frustrating because the more I learn, the less I trust and it makes it exceedingly difficult to know where to turn for good information to use to help my patients. I have always been inspired by CrossFit’s dedication to virtuosity - doing the common uncommonly well. We need more virtuosity in all areas of our world, but especially in research and medicine. I am grateful to at least have some clarity on the basic ingredients needed to build a foundation of health and a buffer against sickness - movement, real food, community, sleep, and relaxation - and will continue to work to sort through the noise that is our biomedical research culture to provide my patients with the best information available.

Comment URL copied!
Nathaniel Pennington
November 8th, 2020 at 6:06 pm

Really enjoyed your comments and perspective as a clinician in navigating the issues in biomedical research and in clinical practice as well.


The unwillingness of researches to turn their back on bad research in light of better evidence was one thing that stuck with me throughout "Rigor Mortis." The sunk cost of years of time spent and millions of dollars marries people to bad ideas and retards progress toward better outcomes for patients. I think it's very similar to many common orthopedic surgeries that are still performed in light of strong evidence that they are no better than placebo. If you are an orthopedic surgeon that does mostly knee arthroscopies and the evidence now says that they might be useless, are you going to be receptive to that if your livelihood and identity is tied to those procedures?


I came out of the book with a more hopeful outlook however. Every chapter dealt with a different perversion of the scientific process but there was always a good scientist trying to find ways to make it better. More transparency, pushes for publishing studies that fail, collaboration between different and competing labs--these things are all great steps in the right direction toward better science and better outcomes for patients.


I was absolutely floored by some of the contaminated cell lines that labs were using. The book hammers home the fact that science is not an abstract idea, it's a process that's carried out by human beings that make mistakes.


Great read.

Comment URL copied!
Liang Kong
November 30th, 2019 at 9:27 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

What 'Rigor Mortis' shows us is only the tip of the iceberg, 'the dark side of humanity.' By not necessary, talking about the 'Seven Deadly Sins.' per se, what we look for here is 'Selfishness.'


The human being, as one of the creatures on this planet, has no different than others in the sense of multiplication, accumulation, exploration, and so on. Let us summarize all of those traits as one word, EXPANSION. It is a part of our instincts, which is essential for survival. However, as modern civilization approach to the current level, people are more reliant on collaboration for their decent lives, here it comes to play as 'Selfishness.'


'Every person for itself' mindset has deeply rooted in the human mind, and this openly represents the dark side of humanity. People are replacing love for casual relationships, kindness for practicality, goodness for hatred, and envy, now what else to remain? Suit individual and particular group's interest by creating manipulated data with bias, leveraging public credibility with sloppy sciences?


In China, people are not typically educated to have a sense of critical thinking in school or society. The average population tends to follow the trend and seek for 'standard answer.' More often than not, those 'standard answers' given by authorities, governments, institutions which supposed to have high credibility are not that reliable as people thought. We use to consider that it was the way in China. However, the world is the same, and some areas are even worse.


We all require to perceive the world by ourselves. We should immerse ourselves with the amount of diligent and intelligent people to explore the reality, the authenticity in practical. Tribe, community, squad, no matter what the name is. It is the group of people that distill by integrity, liability, and accountability. It is the thing that keeps everyone from different corners of the world have truthful conversations here. It is CrossFit.

Comment URL copied!
Katie Hogan
November 30th, 2019 at 8:24 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Since I first found CrossFit.com in 2008 it has been a constant source of continued education for me. With Coach Glassman recommending we read Rigor Mortis I was eager to learn more about the flaws and pitfalls of biomedical research. I consider myself an optimist and when I hear about a particular study or piece of research in the field of medicine I want to believe what I’m told. Prior to CrossFit, with regards to my knowledge of strength and conditioning and nutrition I had most often just believed what I read. I’ve since learned to question everything.


In spite of this I still want to believe that there are good scientists out there doing remarkable work that will bring about major advances in the field of medicine. I acknowledge that what Harris has uncovered is disturbing and should remind us to continue questioning what we read in the headlines, but I also know that it is much easier to point the finger at everything that is wrong with a particular profession without ever having walked in the researcher’s shoes.


Even if scientists in the biomedical field do their due diligence in research and avoid being swayed by greed, ego, or their own lack of patience it is still extremely difficult to execute experiments with the rigor that is expected via the scientific method. When it comes to the fields of biomedicine and psychology in particular there are just more variables in play than other sciences and as shown throughout the book there are complications at every turn. Harris puts it best when he says that “Biomedical research is challenging even under the best circumstances” (27)


Bits of hope are sprinkled throughout Rigor Mortis with mention that many in biomedicine are fighting to bring more rigor to experimentation within the field. There are exciting breakthroughs happening in spite of the more sloppy science that Harris outlines in detail. And while the reproducibility [or replication] crisis is a major issue in the biomedical field, I believe we must be careful not to assume that every researcher is intentionally falsifying data or cutting corners. I want to hold onto hope and believe that failure is a necessary part of science that will ultimately drive innovation. There will always be corruption, but it doesn’t mean that rigor is dead. Just as CrossFit shines a light on what it means to be healthy and fit, we must look for those who uphold the integrity of the biomedical field and work to save lives.

Comment URL copied!
Jongwon Yang
November 30th, 2019 at 3:39 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

My initial thought after listening to Rigor Mortis Audible for the first time was "Seriously?" There are many interesting or disturbing factors that make me no longer believe what the scientists say. Among them, Impacting Factor (IF) of a journal interests me most. Partially because my country South Korea is mentioned. So basically scientists and doctors work really hard to publish a research paper on the renowned journals with high impact factors like Nature, Science and Cell and often get judged by it. And it has become so important they can lose their job over if they are not able to do it. No wonder they can go crazy.

So I started a little bit of digging. I asked some of my clients with PH.D. who works for research labs and universities-unfortunately no bio-related- if this is the case. The answer is YES with reluctance. No matter in which field they work, publishing paper on the journals with high IF or H-Index is everything they think about. A close friend of mine who has PH. D. in engineering said sometimes they know the data they collect has a flaw and the way they process the data has even more flaws but they often ignore the fact as long as the result will grant them governmental funding. "Otherwise we can't survive." That's what he said. Astonishing.

I can't stop connecting this IF to Likes in social networks. There are many good trainers who provide quality training resources with genuine intention. But sometimes I come across with this so called influencers who flesh their six packs and apple bums with some eye catching movements that are almost silly to try. Like some scientists have forgotten their cause for the sake of grant and funding, those influencers seeks for the likes and subscriptions which eventually turn into profit. If that is the world that they want to live in, so be it. But just like many people would believe anything the scientists say on those journals, many people will believe what those influencers say on the social networks. That is very disturbing and dangerous even.

By the time I finished listening to Rigor Mortis three times. I started to look back myself instead of judging other people. And followings are my takeaways as a trainer.

1: Is my training and teaching method valid and proven?

2: Am I not improvising something that even myself never tried before to keep my clients interested and engaged?

3: Am I sharing clients information and their progressions enough with other coaches to help them the best we could?

4: Am I coaching or just running a class?

5: Am I coaching with rigor and relentlessness?


I have a lot to work on.


Johnny

Comment URL copied!
Casey Parlett
November 30th, 2019 at 1:23 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Reading this book reaffirms that modern society has things completely backwards. Even in the medical field that most would like to believe is squeaky clean, is far from it. The constant pressures and stresses that have become placed on self worth, are stripping away the passion in people that could actually be used to make some profound breakthrough's. Instead we are wasting millions of dollars on experiments for people to rush through to get to the next rung on the ladder.

Sadly this same thing takes place in the fitness industry. Everyone is chasing the things that look "cool" in videos or pictures, or the fastest results rather than putting in the work to actually make progress that will stand the test of time. And because people are looking for this, trainers are giving in to the pressure to give it to them and we are ending up with a bunch of non-sense that looks attractive yet is yielding no results.

Comment URL copied!
Lisa Ray
November 29th, 2019 at 11:56 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

As per other’s take away, this read was disturbing. Whether I should not have been surprised or not, it was confronting and dismaying. How is the lack of rigor, the motivation of fear of failure, and meaningless accolades of being published in particular journals so pervasive in fields so in need of precision and accuracy? When the world is in dire need of breakthroughs in understanding and providing treatment and methods of healing, it seems like so many would care more about more precise outcomes? Certainly, there is solid research, but how do we know what is reliable and usable versus all that is not? Who do you turn to? The current societal climate is mistrust of science as it is. These kind of practices and the revelation of them only serves to validate that mistrust.


As with the fitness industry, the healthcare mess, and nutrition myths, it appears to be overwhelming in how one can make a difference. As noted in Rigor Mortis, there are a few scientists and researchers out there who desire to change this tide. To employ rigor to the process. There is good science out there. But how can so few really do anything to make positive impact on these powerful motivators? Ask the MDL1’s. Look to the change of the face of fitness that has occurred over just the last 20 years. Read about the lawsuits and battles that have been won on Capital Hill against NSCA and ACSM, U.S. Department of Health and Human Services, the Centers for Disease Control, the challenge against big soda and #sugarkills. Note the renewed focus on CrossFit.com. What CrossFit is truly intended to be about and who it has always intended to be for. #getoffthecarbsgetoffthecouch


There is a way each of us who aren’t in the field of research, the way a few individuals who are in the field, can make a difference in the face of overwhelming odds. Lead, provoke, and inspire. Individuals, lost in the sea of misrepresentation, misplaced motivations, and negligence, may not be that smart or strong. In groups, we are remarkable. We can lead in our affiliates, provoke others to examine what is in front of them in their own body and life and in the lives of those they care about. We can inspire the demand for change. Maybe the entire mess will never be cleaned up and made right, but small changes have a ripple effect. All those individuals come together in some way, and those ripples make a wave.

Comment URL copied!
Bethanie Giardina
November 29th, 2019 at 11:42 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Thanks for the read, Coach. Harris' book is helpful in exposing how we should consider research we encounter. That is, keeping in mind the funding behind the research, the journal which published it, and what methods were used. Moreover, we need to be mindful of the culture in which research is being bred. I want to see more discussion of this “culture” which seems to be what bred the sloppy science and unwillingness to admit fault in research.


I came across this news article (https://time.com/5709241/open-insulin-project/) while reading Rigor Mortis and it references the researchers who first discovered and patented Insulin as a treatment for diabetics:


"Banting, who shared a 1923 Nobel Prize for his work on insulin, demanded his name not be put on the patent, believing profiteering off a medicine was unethical. His co-discoverers agreed, transferring their patents to the University of Toronto for $1 each."


How have we come from a culture believing it “unethical” to profit from medicine to one where those involved in big pharma, have “big money”? The easy answer to this question is to say “money then wasn’t as big in medicine as it is now”. Sure, maybe, but it must be more complicated than that and, as a culture, is something we need to keep in the forefront of our minds when considering these deep seated issues in biomedical research which Harris highlights.


Harris references potential options for getting out of this mess. One is a call for greater rigor in research from the beginning stages. He also discusses the need for greater “forgiveness” or “lower consequences” when research needs to be retracted. I would appreciate greater discussion of how and if these two approaches can coexist in the current culture of research. Greater than that, how do we change the culture? The answer must lie in somehow removing the culprit for the shift in the culture of science from when Banting made his Insulin discovery in the 1920s.

Comment URL copied!
Todd Occhiuto
November 29th, 2019 at 11:40 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I appreciate and agree with most all of the comments above. It is both disappointing and disheartening to learn of all the issues that Richard Harris laid out in Rigor Mortis. The inability to reproduce results, the lack of rigor, the incentives to publish work rather than to ensure its validity, along with the absence of acknowledgment of potential issues with published results leaves me with little hope in the ability of the science community to consistently provide studies and research that will lead to major breakthroughs going forward.

But as I read this book and continued to hear the issues with most research, I could not stop thinking about the importance of the message that CrossFit, the affiliates, and trainers are delivering to their audience on a daily basis. By tirelessly communicating the CrossFit prescription of constantly varied, functional movements, executed at high intensity, along with eating meat and vegetables, nuts and seeds, some fruit, little starch, and no sugar, while eating them in quantities that support exercise but not body fat, we will be impacting people, improving their health and fitness, and hopefully keeping them out of the healthcare system that is clearly broken.

I finished the book with additional inspiration and excitement for spreading the CrossFit message and doing my part in the battle against chronic disease.

As Coach has mentioned on numerous occasions and this book clearly highlights, there is little hope for the public health going forward. But even still, I know we, the CrossFit community, can continue to spread the message that we are in possession of an elegant solution to the world’s most vexing problem. By doing so we will continue to be the lifeboats in the tsunami of chronic disease.

Comment URL copied!
Camille Acton
November 29th, 2019 at 11:27 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I wonder if part of the lack of rigor in scientific research stems from a systemic societal problem. Focusing on winning over the pursuit of excellence, economics over integrity, and image over substance threatens the validity of all of our results, including scientific research. This hasn’t always been a problem in our culture. There was a time when it was expected that winning,wealth, and image were by-products of hard work, fact finding, and integrity. With the erosion of these values comes the loss of meaningful results. For instance, the increasing acceptance of compromising the integrity of sports may seem inconsequential on the surface, but when this way of thinking creeps into more critical areas of science and governance, the consequences become much greater. The patience necessary for meaningful achievement in science and engineering are at odds with the expectation of immediate glory, wealth, and quick fixes. It seems that scientists, like everyone, are increasingly influenced by twisted priorities. “Rigor Mortis” effectively illustrates the point that truth is bound to be compromised when livelihoods and reputations depend on succumbing to such societal pressures. If we expect ethical scientific research, perhaps we should look deeper into our collective conscience, and seek truth and integrity in our daily lives.

Comment URL copied!
Ashwell Phillips
November 29th, 2019 at 10:28 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis was definitely an eye opener. Even from the preface, I found it quite shocking that only 7.14% of the 7000 known diseases have treatments. I think this just shows us how little we still really know about all that is going on inside the human body.


I found chapter one the most impactful in terms of the message that it got across. It made me think back to my university lectures in ‘research process’ and the importance of validity and reliability. I remember at the time thinking how boring all of it was. Now, coming back to this in Rigor Mortis, you can see why these two principles of research are so important. This was highlighted with Begley’s work at Amgen, where only six tests out of 53 exciting studies could be reproduced. A mere example of wrongdoings that had been going on in the research field.


While this may be a crude example, I thought that the idea behind flawed research designs and the reward for being first in biomedical science could somewhat be likened to that of a gym member who ‘cheats’ during class workouts.

For example, a solid research design is one that has a stable foundation and is easily repeatable. This is your workout of the day - carefully assembled to maximise the power output that any given athlete can produce on the day. In any experiment, it is essential that the researcher follow the specific instructions for how to achieve the desired outcome. Or in CrossFit speak, you need to follow the correct movement order and given repetitions in order to elicit the given stimulus. For the cheat, they will discretely avoid some of the necessary stages in order to come out on top - this being a top finish in the gym, and the attention and praise that goes along with smashing a workout and beating everyone else. And it is only once that person has been called out that they suddenly realise ‘oh shit’, I haven’t taken the necessary steps to make my workout/or my experiment Observable, Measurable and Repeatable.


Personally, I think you can see some resemblance between the two comparisons. But as I said, it was just a crude example. With Rigor Mortis coming to a close, it was great to see that steps are now being taken to ensure that biomedical experiments are now being held to more rigorous standards so that all those affected by disease can be better off. As coaches, it is our mission to uphold the high standards that CrossFit sets, so that we too can affect positive change in those who need it the most.

Comment URL copied!
Benoit Poirier
November 29th, 2019 at 8:59 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I've been letting my thoughts percolate for a few days now and they revolve around three themes:


1: The culture of peer-reviewed science.


My own experiences in university confirm some of the wayward culture I experienced on campus when it comes to how science is conducted. There is a lot of pressure to publish relevant articles with flashy headlines. With money being tight, professors rely on graduate students who work tirelessly in hopes of getting a permanent position, and undergraduate students who work for free in the hopes of getting into a graduate program. A local university even makes it clear when applying to a graduate program that students will be expected to apply for grants and generate funding for the university. One can only imagine how this creates in environment in which there is high pressure to perform. Richard Harris only makes brief mention of psychological sciences, but what he observed easily applies to the field I studied. I suspect, also, that it applies to many other fields.


2: Sensationalist media.


Unscrupulous media reporting can generate a lot of buzz and therefore money around an idea. If the journalist is not well-versed in scientific litterature and the trials and tribulations of experimental design, one can easily be led astray. This in turn leads the public astray, donating money to causes whose methods are suspect. Richard Harris doesn't dive in too much on this topic but there is definitely an issue here. Positive reporting of bad ideas can only further pidgeon-hole scientists into pursuing the wrong path, since they profit from the attention they get.


3: Consumer beware.


We don't, for the most part, teach critical thinking and scientifc literacy in school. We generally "trust" scientists, for better or worse. However, if people knew what they were actually funding with their tax dollars, I'm sure the uproar alone would cause significant changes. I appreciate Greg's effort in arming people with these skills with the critical thinking videos posted here on CrossFit.com, because at the end of the day, only we and our fellow citizens can hold these people accountable.

Comment URL copied!
Jonathan Mears
November 29th, 2019 at 7:55 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Morris

What a depressing book! To know we really haven’t made a significant cure or breakthrough for the major diseases (Cancer, MS, ALS, Alzheimer’s, etc.) since the late 60’s early 70’s is just disheartening. Funny that HIV/AIDS was the last big ticket item we found positive drugs for. Probably because it threatened everyone’s lives upon discovering it was not a “gay man’s disease”.


The level of cross contamination with cells is also hard to believe. On top of that the fact that many labs and scientists don’t double check the cell batch they receive to indeed prove it is just what it says it is is lazy. What happened to “A society formed in Florence after Galileo’s death in 1642 took as its motto provando e reprovando, test and test again.” Pg. 30

One gentleman, Stuart Firestein, seems accepting of all that goes wrong. I can’t even comprehend this notion. Yes, I make mistakes, and learn from my mistakes, but not at the costs of billions of dollars, and people’s lives. If taxpayers knew the results of these tests there would absolutely be a demand for more rigor in testing. “...we need to know why so many published results in peer-reviewed publications are unable to be successfully reproduced. When the NIH requests $ 30 billion or more in taxpayer dollars for biomedical research—which I think is not enough—shouldn’t reproducibility, replication of these studies, be a part of the foundation.” Senator Shelby

I found the chapter on mice being used as test subjects fascinating. When make and female scientist can affect results. The smell of a persons shirt, light, time of day, equipment noise all can change results. It’s hard to believe we use them at all. “... who handles the mice can also make a dramatic difference. “Mice are so afraid of males that it actually induces analgesia, a pain-numbing reaction that screws up all sorts of studies, Garner said. Even a man’s sweaty T-shirt in the same room can trigger this response.”pg 80 Of course, biggest of all is that a mouse is not a human, it can’t tell us anything.

When interviewed 1/3 of all scientist have used questionable methods. Can you imagine? The worst part is they are pressured to do so by the academic foundations they work for. Everyone should read this book as it is YOUR money funding these bogus studies.

Comment URL copied!
Melody-Sara Feldman
November 29th, 2019 at 7:41 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

If you look at any system or corporation, there will always be inefficiency within it. But this system is particularly troubling because what could be more important than the goal of medicine: human health? My favorite chapter of this book was the final one, which suggested a means to at least partially solve this systemic issue within the scientific community, by first returning to "taking technical steps like validating cell Iines, running proper controls…choosing adequate sample sizes…deciding in advance what hypothesis they were testing,” and also by changing the social context surrounding research. Harris suggests that by removing the associated social penalties from unvalidated hypotheses or irreproducible results, the truth could ultimately be revealed at a much faster rate. But the question is: how do we get scientists and researchers to adopt these methods? I don’t have the answer. Maybe someone here can suggest an approach or a starting point?


Until then, I’m going to do my best to work from the preventive angle, as many have echoed so far in this thread. Build the biggest buffer against chronic disease by consuming whole, natural and unprocessed foods in “quantities that support exercise not bodyfat,” and of course, do CrossFit.

Comment URL copied!
Ravikant Dewangan
November 29th, 2019 at 7:28 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This was a difficult book to read at the same time an eye-opening experience for me. What I learned is that there are HUGE amounts of wasted money on useless threads of "research" founded upon untrue priors. The title of this book Rigor Mortis is about the death of rigor in science. Specifically, medical science and biochemistry, This book goes into considerable detail, about why so many research studies are not reproducible.


One of the mind-blowing instances is that we can't even use different lab assistants in some mouse experiments. WHY? Because the mice react chemically very different to male lab assistants than they do to female lab assistants. They are so terrified of a pheromone that they are naturally neuralgic if a male handles them.


Then, there are significant technical difficulties in doing some of this research. Sometimes, the results of an experiment can depend on how a test tube is cleaned, how briskly a chemical is stirred, or how similar or different the genetics are of a set of mice.


Sometimes, the lack of money can be an issue, for example, not being able to afford a verification of the type of cell that has been purchased from a biochemical company, or using a sample of animals that is too small to have any statistical significance.


And, sometimes, experiments are designed poorly. The use of the "p-value" of statistical significance is often misused, and intellectually lazy researchers sometimes formulate their hypotheses after experimenting. Researchers are unwilling to accept that a hypothesis is wrong, even after it has been proven to be incorrect.


It is sad to see that the physical components and culture of the scientific method no longer give valid results that are reproducible elsewhere.


Harris proposes four areas that scientists community agree are necessary to fix the mess of academic research:

(i) getting scientists to change their research habits

(ii) changing journal incentives

(iii) better practices at funding agencies such as the NIH and DoD

(iv) forcing universities to grapple with these issues


On the other hand, the fix that most appealed to me was the idea of just slowing down and work more deliberately, with exceptional care given to the experimental design. This mentality of rapid progress at all costs permeates modern society, and I think all we all would benefit from slowing down.


Regards,

Ravi

Comment URL copied!
Arianna Sisto
November 29th, 2019 at 5:28 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Money- corruption- greed.... it is history repeating all over... did we think it would be different in a field that should aim at saving people’s lives, that should find a cure for chronic disease?

Unfortunately that is not the case, and some of us had to learn it the hard way, watching friends and family members fade away while doctors could not explain why the cure was not working.

Rigor Mortis by Richard Harris has displayed a very important and complex topic in an accessible way to most. The vast number of case studies he provides, make it indisputable that some modification is needed in the biomedical research field and that it is imperative that new quality standards are set, new incentive systems that aim for quality rather than quantity in order to achieve accountable results. Hopefully this book will strike changes within students and academics in the biomedical research fields, push them to share data and methods, make science more transparent and hence more reproducible. Possibly, it will trigger the creation of new incentive structures, rather than ‘ publish or perish’ and reinforced federal laws.

Do I believe this will be done? Probably not. Is it possible to go back to Darwin’s times where there was no hustle for money but publishing discoveries was mostly for personal pride? Definitely not.

Do I have the tools to identify fallacy in scientific studies? Unfortunately not. But what this book has made me realize even more is that Crossfit is heading in the opposite direction, the “cure” is to help people stay away from chronic disease and lead them to a healthy and balanced lifestyle. Fighting this together is the real purpose of what we do.

Comment URL copied!
Federico Biasetti
November 29th, 2019 at 4:04 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

One the things that I’ve noticed in many, many fields is lack of professionalism. Very few professionals of all the ones I’ve met during my life made me say: “that guy knows what he is doing!” or “finally someone who knows how to do his job”. And I think sloppy science is based on that. Lack of professionalism. And often times, I believe, lack of professionalism is based on lack of care. This lack of care caused a domino effect of studies based on “bad studies” which is costing a lot of money and most of all, not curing diseases.

CARE is what is needed, in every field, in every profession, in everything we do.

Comment URL copied!
Meagan Burns
November 29th, 2019 at 3:32 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

The resonating theme this book left with me was on education; the more we educate, the more change we can make. This is very similar to the world of nutrition. At the Level One, we provide data driven research that doesn’t necessarily make a huge, immediate dent in the chronic disease epidemic, but we are getting the education out every weekend and we are changing the world. The Level One kernel and book commonalities are, “long term commitment to small steps is better than no change.” I believe this is what Harris is attempting to do with his book Rigor Mortis. Now, “we” as a community can help educate others and spread the word of the “mess” in science.


At the CrossFit Trainer Summit, we had the pleasure of listening to Dr. Nathan Jenkins, an exercise scientist, who is an example of these small changes happening in science. He is brave enough to go against the grain of what his peers in the field are doing because he believes that CrossFit can change the world.


I am grateful to work for a company that prioritizes health vs. monetary gain.

Comment URL copied!
Cody Keathley
November 29th, 2019 at 3:27 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I thoroughly enjoyed this read. I appreciated the enlightening aspects and attributes that make up scientific studies. I was shocked at many points during the book to read the lack of standardization and scrutiny needed to present a valid finding. In particular, I appreciated the example that paints a picture of how experimental mice are relative to humans. The example outlined that the use of mice would be similar to using only “35 year old white women, from the same small town, with identical husband/diet/environment/IQ and have the same grandfather.” The use of such a homogenous sample group makes no sense or correlation to the population as a whole.

Comment URL copied!
Dan Hollingsworth
November 29th, 2019 at 3:21 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I found Rigor Mortis to be strangely fascinating and equally, or perhaps more so, frustrating ... on so many levels. Perhaps the most frustrating part of it for me was the lack of integrity on the part of many (not all) in the field of biomedical research. The inability to accept the fact that perhaps you were wrong about something is unconscionable to me. I understand that when you invest a great deal of time, thought and money into something that you have a strong emotional bond to it, however, how can someone go into this field of work and not be open to challenges and debate? That defies logic, in my opinion.


I couldn't help but draw parallels between biomedical research and politics. Harris describes the stereotypical path of a researcher as someone who starts by wanting to "change the world for the better", but then soon becomes the researcher who becomes afraid to push boundaries or challenge colleagues for fear of losing grant dollars. This sounds too much like American politics where freshman politicians come in shaking their fists and are soon left chasing only re-election votes and toeing the party line. In the end, it is we who are left holding the bag and dealing with the consequences of a status quo mentality.


My hope is that the number of scientists who are open to having others analyze and challenge their methods and findings will grow. Perhaps research dollars should be parceled out and ultimate payoffs are held back until a study is found to be reproducible. As others have already said, the consequence for "sloppiness" is too high. Something needs to change.

Comment URL copied!
Todd Widman
November 29th, 2019 at 3:06 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis


What a privilege to work for a company that educates, and further holds in such high regard those who are of character and the right ilk over what formal schooling and letters might be behind one’s name; thank you, Coach, for creating the CrossFit book club.


I was filled with apprehension and concern when I started reading Rigor Mortis; how bad is it? I felt myself pulled by two extremes, on one side I am the younger brother of a brilliant MD and saw firsthand how hard he worked in medical school for half a decade after I graduated from college, added to the fact that I am unabashedly dependent on modern science and medicine to provide comfort and care for my family and friends when they become sick or broken. On the other side, having the privilege of CrossFit and Coach’s guidance for just short of 15 years I have had a front row seat to the exposure of inaccuracies, half-truth’s, lies, and deceit in Nutrition and the Strength and Conditioning field, so unfortunately the title page outlining sloppy science wasting billions was far from a shock.


I found Richard Harris’s book to be a quick, easy read, full of a balance on both sides of the coin, written in a manner digestible for laypeople such as myself. I especially appreciate his opening acknowledgement of how hard medical research is (1) as well as that scientists face a choice (like all people) to do the right thing, or the thing that they perceive is necessary to maintain a career (3). His preface ended with the nod again to the unbelievably difficult reality that most of science is built on inference rather than direct observation (4).


From there I buckled in for what I consider to be a 10,000 foot view of the myriad of ways in which scientific research can go wrong (using old studies, using false studies, cellular adaptation, animal adaptation, differences between mice and humans, differences in how scientists use laboratory equipment, etc.) and how scientists can go wrong (money, greed, ego, promotions, tenure, laziness, false assumptions, lack of checks and balances, biases, etc.). Not a scientist, I found this illuminating; there are so many ways in which the science and the scientists can be led astray it is almost a miracle anything has gone right. Add to that the reality that “some of the most successful drugs are a result of serendipity…(90)” and I am only further amazed.


Crooked people, crooked methods, crooked science…what to do? On completing this book I find myself only further inspired to be working for CrossFit and beside the incredible HQ staff; we are trying to right the ship that is undeniably listing. I also find myself keenly aware of how I speak… though I love the comfort and stability of absolutes, that is often not reality; the smartest people I have listened to and read often add “what I have found” or “research would suggest” to their statements. I shall try to do the same. Here’s to experience and wisdom, thank you again for the excellent read.

Comment URL copied!
Jennifer Hunter-Marshall
November 29th, 2019 at 8:26 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Such great discussion. I have enjoyed reading all responses and I struggle to find anything new to add. Rigor Mortis, was not an easy read for me. Initially, I was left feeling disappointed and helpless. After some time and more reflection, my feelings shifted more toward hope and action.


I admit I was very naive about the goings on in biomedicine. I assumed that most studies being done were sound. As the daughter of a breast cancer survivor, I needed to believe that the institutions and scientists that we hold in high regard were continuing to make progress. I blindly trusted.


As Harris points out, practices haven't changed because that's the way they have always been done. If we keep repeating this vicious cycle, nothing will be accomplished.


I am so thankful to those that are shining a light on this topic. I am thankful for the opportunity to be apart of this community of thinkers and doers. The hope I now feel, comes from knowing. You can't fix something if you don't know it's broken. We need to chase truth using the best practices because lives depend on it. Every day.










Comment URL copied!
Nicole Gordon
November 29th, 2019 at 3:51 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I'm writing on Thanksgiving evening after a wonderful day filled with great food and great conversation. A discussion earlier today turned to family and friends of our parents who are sick from disease (cancer, alzheimer's, multiple surgeries, etc.). My mother-in-law declared, "they need to find a cure for cancer!". My husband replied that the problem is there are so many different kinds that the task seems nearly impossible. He also noted, however, that people are living longer (or in remission) more now than in the past (as compared to great-grandmothers who both died of cancer). I told my mother-in-law about "Rigor Mortis" by Richard Harris. I explained that according to the author, there are many problems facing biomedical research and how the lack of reproducibility has led to a loss of millions of dollars in a trickle-down effect. I also explained that prominent scientific journals are eager to publish "breakthrough" findings quickly for the promotion of the journal, the researchers, and those who funded the research. She asked, "So, you mean they think it's more important to be exciting than right?"

She sounded sad and angry at the same time. I completely understand and feel the same. It's hard to read about eager young researchers that are "born" into a community that has promoted a haphazard research process with little oversight on proper execution of techniques. I was floored to read about the exciting blood test for ovarian cancer (previously only diagnosed through surgery) that was promoted only to learn that the results were not actually reproducible. The "data" was shown to be nothing more than "noise", but the researchers chose to use the data that agreed with their goal of protein detection of women who had cancer vs those who did not. How and why would they choose to do this? I'm grateful to have had the opportunity to read this book. It just shows me that I must continue to ask questions and to spread the message to dig another layer (or several more layers) deeper.

I felt the positive side of this book too. I heard the "hope" that with this book's message that change is possible AND is slowly making its way through the biomedical research community. Richard Harris notes that the slow but steady progress towards "rewards" for meticulous and thorough research is coming. The recognition of the limitations of using mice, rats, or other animals for human studies should continue to be taken into account. If scientists are able to state the limitations about their results and/or the processes used to obtain their data, more informed decisions will be available.

Needless to say, we had a positive discussion about this book. It took us to the broader discussion of health and wellness and the need to continue to do the things that we know will keep us away from sickness no matter what the "breakthroughs" say.

Comment URL copied!
Jason Ackerman
November 29th, 2019 at 3:22 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Echoing the sentiments of people I highly respect, the trainers of the CrossFit Seminar Staff, I'm embarrassed. I'm embarrassed because for so many years, the words, "peer-reviewed" and "scientific data" actually led me astray. I followed and believed in studies that weren't true and worse, I wasted time and energy doing things the wrong way.


Health Care and the systems that surround us have led us astray and done us wrong. They are taking advantage of us. Costing us millions upon millions of dollars and more importantly causing so many to suffer. Why? For more funding or to further their careers?


The same holds true in the world of fitness. Top organizations are out there lying in big, bold, expensive ways, and people are believing them. Like me, why wouldn't they, why would they be lied to? Even if they knew, where else will they turn?


It's our job as leaders in the community to educate. We see how easy it is for friends, family, loved ones, to go to the doctor only to be told they need to eat less, or watch their saturated fat, or deadlifts are dangerous, or any other nonsense that is being spewed to further their bottom line.


But as leaders, we need to constantly be in the pursuit of knowledge, we need to question the status quo, and be open-minded. Open to the potential that we don't know everything and we aren't always right, but when we're not, we find out why, and learn from it.


Comment URL copied!
Edward Morrison
November 29th, 2019 at 1:01 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

As I made my way through Rigor Mortis I found myself experiencing both a sense of disappointment and optimism in what the book presented.

Like many have already expressed I was shocked at the lack of uniformity and several cases of negligence that Harris puts forward. The disappointment is two-fold, as so many of the public and professionals rely on these studies being accurate to drive correct behaviors or the manufacturing of correct drugs, who or what can we trust if we know that so many studies are the victim of a hidden agenda, personal gain, lack of energy or just general sloppiness? Additionally, a concern was for those researchers who do toil endlessly to dot the I's and cross the T's in a bid to find the truth. Dr. Heinz & Ying Sue references in chapter 2 who for over a year to figure out why they were getting different results in an experiment that seemed almost identical only to discover it was the way they were stirring some of the cells in a flask. This type of attention to detail is drowned out by the many other cases in the book that don't display the same rigor.

So what is there to be optimistic about? As a CrossFit trainer for 10 years, I've always had to deal with the unnerving reality that what I was seeing transpire in front of me did not necessarily match the readings of my textbooks. I was doing things that studies suggested I should not in relation to energy systems, functional movements for elderly populations, lowering people's carbohydrate intake just to name a few, and all with great success.

Rigor Mortis and it's unpacking of the problems with scientific studies make me more committed than ever to trust that as long as I can conduct my role as a trainer with rigor then the people I serve will always be better for it.

What is Rigor as a CrossFit trainer? To me, it is doing your due diligence in terms of acquiring knowledge about all things health and fitness but more importantly, it is accurately and honestly assessing what results you are getting from the input (our input is constantly varied, functional movement executed at high intensity fueled by meat, vegetables, nuts, seed some fruit little starch no sugar) without any agenda except for maximizing the results for the client, which is their improved health and fitness.

When you see the results, adjust the input accordingly to get better outcomes.

Comment URL copied!
Jonny Davies
November 29th, 2019 at 12:22 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

First of all, how cool is this book club? this was not an easy read (listen) for me, on a rainy drive to and from Manchester. Way outside my area of expertise, but here are my takeaways:


My mood moved from realising how hard I would have to listen to the content, to a bit of disbelief in how easy it is for research to go wrong/be misleading, then finally to realising that CrossFit does not have these same problems, and in fact a perfect example

Of science done well.


It first grabbed my attention when the author mentioned that only a few in 100 studies could actually be reproduced, and how many factors could be the reason for this, Money, a race to be published, bias towards the results they want to see, a lack of transparency, no standards to the methods being used,

Then about half way though I was pretty blown away by how much a study's results could change based off on what is used to clean a test tube, or whether the chemicals in a test tube are Shaken or stirred (007)

Kind of made me think well how do we know anything? My mood improved when he talks about certain people standing up and trying to do things right ...


Then it made me reflect on CrossFit , how right we get it every single day. we define our terms, and we prove them right over over and again, all over the world, every single day.


times going down? Rounds are going up? Loads are going up? Eating Better? sleeping well? Community of friends suddenly getting a lot bigger?

Then your becoming a fitter healthier happier human being. But we already knew that.


Looking forward to book 2




Comment URL copied!
Jonathan Gary
November 29th, 2019 at 12:02 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

First, I appreciated the author admitting to the hyperbolic title in the preface. Additionally, I liked that he delineated the complexities inherent in conducting biomedical research; that yes, there are simple (as well as complex) procedures to make everyday experiments more robust. But he is realistic that even these changes would not eliminate the dizzying amount of noise and/or mistakes that could still occur and, at worst, could delay advances. With each problem the author identifies, he deftly describes how there is no simple (or one) fix; not unlike many other large scale efforts even outside the scientific realm.

Overall it was a simple narrative plainly stating the observations that: biomedical scientists are human beings with the inherent foibles, biases, drives, and incapacities; pharmaceutical companies and journal publishers are corporations with competing responsibilities to shareholders and consumers; academic institutions use metrics of quantity (or specific journal) over quality to assess candidates and faculty; government agencies are ill-equipped to administer the funding and regulatory responsibilities required to not only promote insightful research but also allow effective treatment of constituents with various diseases and illnesses; and finally, that research (big or small) on complex systems, is, well, complex.

What to do? For example, as an individual: will you eschew biomedical science completely (i.e. because it’s fallible, therefore it follows that it cannot be used to support or deny any position or action at all); do you have the education/knowledge to ascertain the validity of biomedical research to determine which results are true (i.e. statistics, biology, physiology, biochemistry, etc.); will you trust others’ determination of valid biomedical research results (whose? what will be the metric to decide? and keep in mind the possibility that sometimes the consensus or current truth is/has been wrong and other times the skeptical position becomes the consensus or new current truth); maybe you will just do your own research and exist off only those truths you determine. How a society views and uses biomedical science is, obviously, an even larger question but the same questions exist. And instead of the consequences being contained to an individual, the repercussions now include many, many others. What to do.

Comment URL copied!
Antonio Aguirre
November 29th, 2019 at 8:33 am

I totally agree with you, Sir. The main point can be summed up by just one "simple" thing: scientific literacy.

Comment URL copied!
Jacqueline Aumeyr
November 28th, 2019 at 11:57 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Albert Einstein once said “If we knew what it was we were doing, it would not be called research, would it?” As Einstein aptly notes, researchers are indeed trying to chart new territory and therefore there is an innate element of cluelessness and discombobulation to the work. However, cluelessness and discombobulation in this context is not normally considered to extend to the quality of the practice of science or the quality of the professional climate in which is it conducted. Rigor Mortis by Richard Harris delves into the current state of biomedical research focusing primarily on the downfalls of current practice. It does a good job to expose (basic) problems relating to every part of the scientific cycle, from funding and inception through the publication and consumption of biomedical research. This book had me continually oscillating from shock and dismay to apathetic support of opinions I already had with regard to science. In the end the book did not leave me angry or optimistic - the book left me with questions of what does this mean for me personally? Professionally? And for those other lay people who I come into contact with? Especially, in the midst of a war on facts, from what feels like every angle?


I appreciated how early on in the book Harris lays out how the forefront of science is a nebulous entity who’s nature is continually to prove and disprove itself. It was good reminder of what the end goal is. So, I was not surprised about the amount of contradictory date or differing viewpoints between scientists. I was not surprised by other things. Given my knowledge of current funding of science, I was not surprised as Veronique Kiermer put it, “It doesn’t necessarily pay to be right. It actually pays to be sloppy and just cut corners and get there first.”(p.172) I would even argue one step further that it pays to create the scientific results people want to have. I was not surprised a as Brian Martinson expressed in the book (p.180) that most scientists are working at the absolute edge of capacity and when they reach their limit what is left is to cut corners to finish the work. After reading the book, I spoke with my husband who was chemist and his school friends who are practicing PHDs in their fields. They had all been worked beyond the bone. However, they were and are all very passionate dedicated people who hold themselves to high standards and are genuinely trying to do something positive with their research.


In this regard I felt very neutral toward the book and towards science. There are both exceptional and crap people in every profession. I would not write off the entire scientific community as corrupt or lazy or ineffective. Also, sometimes shit just happens. While Rigor Mortis is primarily focused on the downfalls of current system it also provides examples of positive changes people are tying to make and through each person interviewed shows how there are very dedicated people out there in the field who want better. In these parts of the book I felt some empathy and felt comfortable in my already held opinions. However, this only encapsulated half of my response to the book.


I did also experience shock and dismay at points in the book. To have a concern about scientists’ understanding of “statistically significant” to the point of the American Statistical Association having to address the meaning was shocking. To have examples of how poorly the scientific method has been and is administered was shocking. I was definitely someone who held a reverence for science as the pinnacle of accurate, methodical, sound research. Many of the mistakes highlighted in the book were so simple. On page 164 there is a quote from Arturo Casadevall in which he states, “Scientists need to be taught to Think. We can teach scientific thinking, but we don’t do it in a formalized way…we need to teach scientists how to design experiments properly.” I was truly taken aback by this. What an audacious idea that scientists need to be taught to think, wasn’t that what they were supposed to be doing? Wasn’t the scientific method and how to set up an experiment one of if not the first step of that? These were skills I learned early on in school, how could they not be practiced by those they are named after? I took a moment to think more about this. I contemplated how seriously I took all scientific endeavors in school. Then I thought about my classmates. I must admit that more often than not, especially in a university setting I was disappointed that classmates and group project members had a poor ability to understand and address the question of problem being set before us by our professors or to create and test and idea. Additionally, these students still got reasonable grades. I assumed that that would not be a trait shared by the science department. That turned out to be a poor assumption. So, this makes me think greatly on the education system, but alas I will leave that for now.


Going into the book I felt like science was well poured cup of beer, maybe a little on the side of the glass and just a nice reasonable layer of foam on the top. In this, the foam is the ever evolving part of science and the beer is the solid base which it has built and from which it stems. However, after reading it I feel that the beer is more like one poured by a wasted kid at 3am – liquid strewn all over the place, very little beer, and lots of foam. Now I am left with more questions. I am not practicing in the biomedical field, so what do I do with this information? Personally, how do I sort through the science out there in a meaningful fashion? In the book, John Ioannidis sets out to go over a vast sea of genomic papers to see how many stand the test of time. He found that only 1.2% of studies do. The book also addresses the problem of the volume of work being put out. So when Ioannidis looks at a sea of work he is probably only scraping the surface of the ocean of work out there. How do I work with that volume and that poor percentage rate? How do I account for the pitfalls touched on above? How do I engage with others about scientific research? Especially those who are less interested or knowledgeable in the area that I passionate about? When there is a war on facts at every turn how do I talk about science without demotivating or destabilizing someone more? I have not answered these questions yet. I do think the idea put out by others that the pursuit of excellence is choice. If we’re excellent, maybe we can create a better consumer base that demands excellence trough a different channel that ends up influencing the science.


P.S. I love that Harris also takes that time to mention that there is no magic pill to cure obesity in the book. Thank you for putting that in!

Comment URL copied!
Joe Alexander
November 28th, 2019 at 10:15 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

A wild ride for a book about biomedical science. Hats off to Richard Harris for making a very difficult subject accessible to a layman like myself. I particularly enjoyed the education on “six flags of suspect work,” the impact of bias, reproducibility issues, contaminated cell lines, the batch effect, P hacking, the Texas sharp shooter fallacy, HARKING, and the file drawer effect.

I found myself reflecting upon my own research experiences during my independent studies as an undergraduate studying drug interactions using rats. I immediately recognized many sources of potential lack of rigor. Did we all truly follow the same procedure when making electrodes? What about the differences handling the animals? What was the variance in each animal’s tolerance to anesthetic and the anesthetics effect on the CNS? I imagine there are many additional unforeseen variables (that I can’t articulate) that could impact the firing rate of the neurons we were measuring that were potentially not factored into the equation, especially when your data collection team is a handful of undergraduate students.

This was a great selection to lead the charge in pursuit of the truth amidst the mess. If anything, this book really highlighted the trials and tribulations of conducting sound research, and the ethical responsibility to get it right. Thank you for the recommendation!

Comment URL copied!
Brett Fforde
November 28th, 2019 at 9:42 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Hell in a hand basket? Richard Morris did a great job of explaining that science, good effective science is hard and it is not being done very well in a lot of cases. Not an easy read for someone who would rather hang his hat on scientific principles rather than religious dogma. It makes me realise that at some point there is a requirement to trust and have faith in a high power exists in both situations.

At a base level science holds universal truths that can be witnessed with our own eyes. Whether it is gravity or cell mitosis science has that stuff pretty worked out. As we drift into trying to understand more complex systems it is to be expected that it takes longer, contains much more error and will not be as straight forward. The weekly preaching that we deliver with respect to the scientific method is to be rigorous in your experimentation on yourself. The only news that is definitely not fake news is that which you witness personally and similarly the only science that no one can shake your belief on is that which has been expressed in a cause and effect way on you. We explain the black box model, something into the black box leads to something out of the black box. Inside the black box stuff is happening and we might not understand it but it doesn’t matter if we know the inputs that lead to the desired outputs. Changing the inputs from time to time to see if there are better outputs is part of the fun. These changes can be inspired by convincing theories or ideas in an effort to address the curiosity that a better outcome is possible.

Comment URL copied!
Matthew Chenard
November 28th, 2019 at 7:25 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It was overwhelming and disheartening for me to read about how studies are not always reproducible. Knowing that C. Glenn Begley took 53 studies that he viewed as ”groundbreaking,” and was only able to reproduce 6 is not necessarily a reassuring statistic. However, as Richard Harris explains, there are a lot of different reasons this seems to be the case.


What intrigues me the most is the human nature aspect of it all. Humans want to win. As I read further along in the book, I began to see how unsurprisingly, researchers are incentivized by money, prestige, and publication of their work, not by the legitimacy of their research. These pressures and expectations seemed to be the factors that determined if they had long “successful” careers in their fields. It doesn’t look like it pays to be wrong. I think the problem is that successful studies aren’t seen as important because of what they are telling us, but as something that determines the winners and losers. The idea of winning and losing puts added pressure and incentive on the researchers to produce results however they are able to; right or wrong, good or bad. As a result, the research industry as a whole suffers.


To spend your life’s work on something you are passionate about, and to never see the fruits of it in your lifetime would be a hard pill to swallow. Maybe so. But then I wonder… If spending time, energy and money for something to fail would still be considered a success if it leads future generations to discover the truth. If the only thing humans continue to celebrate it the end goal, and not recognizing the purpose of the process, nothing will ever change. Humility and servanthood need to be the bigger cause in order to make change and drive science and research forward. Seeking publication, money, and fame only seeks to serve our own self-interest and not the actual goal of helping people. But, that is what is shown to be valued.


As Harris states, it is hard to change a culture. It is not our nature as humans to showcase failings or shortcomings. However, this seems to be the only way that this industry will progress the bigger cause. Just like any institution, business, or relationship, things don’t progress if we only look out for own self-interest. The goal shouldn't be to have the most successful number of studies​, but rather live into the bigger mission. Having a finite perspective creates the perception that we can fix it all in our lifetime (and get the credit for it). Thinking past ourselves and instead to consider generations to come, we will be enabled to fix the process and not just the methods. This is a lofty goal. Everyone wants to be right and only a small % of people are okay with taking a back seat to further a noble cause.

Comment URL copied!
Carlos Fernandez
November 28th, 2019 at 7:04 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I admit that I was unaware of the details of the sloppy science that seems to be the norm in biomedicine. Now as I write this comment, I imagine that if I was unaware of this situation then I’m willing to bet the general public is as well. Could the effort of making the general public aware of the waste of money be the first step to reverse the lack of rigor in the research?


As an ambassador of CrossFit I give thanks to Coach for encouraging us to expand our knowledge and begin to invest more time in bettering ourselves as a trainer who go against the grain. Being aware is the first step of being able to make a difference.


The book demonstrates how the funding, the peer-review system, and the need to be the first to publish are just a few reasons why the quality of research is no longer the objective. This is why we are in a reproducibility crisis.


Clearly scientists need to have incentives wether it be money, published articles in a prestige journal or just the satisfaction of potentially helping lives. If these incentives are driven and reinforced by their quality, scientists may begin to work towards a higher standard and not allow for unethical or lazy research to be present.

Comment URL copied!
Matt Lodin
November 28th, 2019 at 6:43 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Thanks Coach, for the recommended reading. Excellent book outlining an overwhelming problem.


While I did conclude this book with feelings of disappointment, anger, and cynicism at the scientific community, I also can’t help but feel renewed pride and hope through the CrossFit community. CrossFit, for me, continues to be a shining beacon in this “mess.” (as Coach so elegantly calls it)


While this book outlines how incentives and systems are almost designed to bring out the worst in researchers, everyday inside the affiliate I see the opposite. Everyday, the workouts we do, the way we eat, and the community we build is enriching the lives of those around us.

Comment URL copied!
Peter Shaw
November 28th, 2019 at 9:35 pm

Renewed pride for sure, Matt.

Comment URL copied!
Ricardo Mendes
November 28th, 2019 at 6:34 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I have heard coach Glassman talk about “the mess” and I had an idea that things were bad, but by the beginning of Rigor Mortis I was scared. Scared because like most people I expected too much on science and prior to reading this book I believed scientists would uphold strong standards and would dedicate their lives in being right.


The thing is that the scientist I imagined doesn’t exist because “Science doesn’t happen in a social vacuum” and people are building careers that depend on publishing papers first to keep the money coming. There is something really wrong in the biomedical research and the “rat race” this scientist are in discourage best scientific practices. Medical science should be applying biological and physiological principles to clinical practice, instead there is a comercial motivation to cut corners, bend the data to the best possible light, not share their work and a lack of transparency that shows the entire Biomedicine culture needs a serious repair.


I am from Brazil and we look with admiration (almost devotion) to American Universities and it was disturbing to realize that no one is teaching the scientific method in a formalized way, even the best possible places. The author describes errors with animal models and untrustworthy cell lines but more annoying was seeing that mathematical and analytical problems are more impactful in the reproducibility crises we are in. Scientists are pulling out the bits that support a hypothesis, and ignoring the bits that don’t. Looking for “statistically significant” instead of something meaningful.


The biggest take away from this book is that we all should expect less from science. “Science is provisional and should be treated as such”. It is absurd to think that with all the billions of dollars invested, there is no progress in the treatment of Glioblastoma type of cancer in the last 100 years, and this is just one of many examples. If we lower our enthusiasm a bit, scientists will be less likely to run after dubious ideas and our clients less likely to embrace the latest dietary fad.


As a Fitness professional I was able to made some connections with the problems we see in our field and the sloppy and unmanageable scientific literature. The lack of dedication to the basics and a focus on quantity over quality brings ineffective results and wasted resources in both areas. Fortunately we have CrossFit and coach Glassman’s support to have discussions like this. He has changed the fitness industry with the first rigorous scientific definition of fitness and now he is providing us the first step to change medical science, making sure that the problems and bad incentives are well understood by making the public aware of Richard Harris brilliant work.


If we are going to be able to change the whole biomedical science field culture depends on figuring out how to create new incentives for scientists and universities and if in the beginning of the book I was scared, at the end I was hopeful because I saw we have good people trying to do the right thing. Right now, reading this comments and seeing CrossFit facilitating this content being more accessible to the public, the hope of getting things right is higher.


Thanks Greg Glassman and thanks CrossFit for bringing this very important matter to our attention. “Doing nothing is not an option” so let’s keep doing science with measurable, observable and repeatable data and changing peoples lives telling the truth for a living!

Comment URL copied!
Andrew Edwards
November 28th, 2019 at 5:50 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I found the content of Rigorous Mortis fairly compelling in regards to the current climate of quantity over quality in the biomedical research community.

In an area as important as biomedical science where a breakthroughs could literally change disease for populations, the industry is falling short in taking the due diligence needed.

I think most who picked up the book initially had an understanding of how peer reviewed literature and research can be slightly off the mark or used to argue certain agendas, but I wasn’t aware of the wild negligence of scientific rigour. While there are a few shining lights trying to break the dichotomy unfortunately they are experiencing a “ resistance is futile” from their peers like the crew of the Star Trek Enterprise on being assimilated by The Borg!

Comment URL copied!
Js Smith
November 28th, 2019 at 5:16 pm
Commented on: 191128

Turkey trot time. 35:12, pretty good time for me.

Happy thanksgiving, y’all!!!

Comment URL copied!
Jennifer Charlesworth
November 28th, 2019 at 5:07 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

In a time where the Google machine can lead us down a rabbit hole of information, I was thankful for the honesty of this book. It leads me to second-guess everything I read! I don't think that is a bad thing, but if so many studies are misleading or leaving out important information, what can we actually trust? I recently finished two books that have a similar tone...debunking old research and/or bringing to light how current guidelines for pregnant women came to be and how/why the research is flawed for those current guidelines. When we take a closer look at the details of scientific research, it seems like common sense that we wouldn't trust many of the studies. I don't know about any of you, but I don't really have the time or energy to dissect every single study about topics that matter to me. I felt a little disheartened after reading the book but thankful for the awareness to not accept things at face-value.

Comment URL copied!
Andrew Charlesworth
November 28th, 2019 at 4:53 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I am very happy that this conversation and book club was highlighted during the "Game Changers" documentary craze! Change my mind but I was not a fan of everything that went on in that documentary and how I had to answer questions in my own affiliate. Here are some thoughts i had with reading. There’s a huge disconnect between scientific research in the business of medicine and the biggest flaw is that there’s no glory or money in checking other peoples research that scientists use for studies


One of the problems highlighted in the book is how too many people are also doing research. Perhaps a solution might be to have a trusted source that’s focused on each specific chronic disease and they work as a team to research the studies and then evaluate go back and check and see if the results are consistent. Once other thing that was interesting is the inability of said research to be re-created. If a study is not measurable, observable and repeatable than it simply should get filed underneath "opinion: and not science.

Comment URL copied!
Jim Rix
November 28th, 2019 at 4:51 pm
Commented on: 191128

For time, with 80# (Rx is 115#)

10 shoulder presses

15 overhead squats

20 push presses

25 front squats

30 push jerks

35 back squats

12:04


Happy Thanksgiving to all!

Comment URL copied!
Keith Wittenstein
November 28th, 2019 at 4:45 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

While reading Rigor Mortis by Richard Harris, I felt overwhelmed. The scale of the problem with science was far greater than I would have imagined. I was quite aware of the "publish or perish" problem that is based on monetary incentives. I knew that in the scientific field it was often better to be first than to be right. I was aware that there is a large replication crisis in science. However, I was dismayed to found out how almost useless mouse studies are. I was horrified to learn that an alarmingly large number of cells that are used for in vitro testing are not the cells they purport to be. And chapter after chapter Harris outlines additional problems with modern science that were hitherto unknown to me.



I was left asking, "why?" Why is science so screwed up? Certainly there is a problem of incentives where good, well-meaning people are pushed to publish studies that are novel and not necessarily right. They are not incentivized to correct the current system. However, what I find even more disturbing than the monetary incentives is the problem of science by consensus. Science's hands have always been tied by the majority rule. Everybody agrees that high fat diets cause heart disease therefore it must be true, right? If you claim otherwise, no matter what the science says, you're a heretic.



This problem has only gotten worse because now we live in a time where we are divided by science in the same way we are divided by politics. You are either high fat or low fat. You are either plant-based or carnivore. You are either a environmentalist or a climate change denier. Science should help us resolve these debates but science is ultimately failing. I cannot help but think that the more debate there is over a subject, the longer some of these scientists will have jobs continually stirring the pot.



Another shocking thing was reading that the drug companies "waste" billions of dollars based on faulty research. Harris points out a common scenario where a paper is published claiming some new serum x cures dyslexia in mice. (My hypothetical, not his) The drug companies read this study and see dollar signs. They rush to try and patent serum x and spend billions trying to develop and take serum x to market. Unfortunately, the original paper was based on bad science and fatally flawed. The drug company finally learns this after spending billions. Certainly if the drug companies knew this they would put an end to this problem. So why don't they? It occurred to me that this is its own industry. Those billions of dollars are just getting passed around a self-contained eco system that funnels money from us regular people that pay insurance premiums and buy medications into the industry that is just perpetuating their own existence. It is maddening.



At the end, I am left more cynical than ever about science and medicine. Nonetheless, I believe that if more of us are skeptical and know more about how to ask the right questions and look for the potential biases in science we will be better for it.

Comment URL copied!
Simon Jones
November 28th, 2019 at 4:42 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

An engaging but hard read (well listen). Some parts were very eye-opening but not at all surprising in terms of trying to publish as many papers as possible to establish a reputation to get more funding to push more research. The more I listened, the more you saw the Finite mindsets or objective-based thinking behind it. It was a race to find a solution, while, as Harris mentioned, there is nothing wrong with that, but the framework behind it meant that tests were not as rigorous as they could have been. Corners cut, samples contaminated, and findings were not double-checked to make sure the right conclusion is met. Then based on that, more research was done based on "bad science'.


I loved his solution, and its a very similar solution, Coach Glassman prescribes. "The pursuit of excellence." To do things the right way, it may be slower; it may take longer to find the correct conclusions, but it's worth it.


I felt this translates to so many other things. Coaching, training, life, the pursuit is simple. Be better! Hold ourselves to a higher standard. We are in an infinite game. There is no finish line, but the longer we can stay in the game potentially, the more chance we can do for the better.


"Doing the right things for the right people for the right reasons."

Comment URL copied!
Margaret Landry
November 28th, 2019 at 3:36 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

"Screw it. Everybody cheats; I'm going to cheat too"

"Scientists are smart, but that doesn't exempt them from the rules that govern human behavior."


What an interesting look into the complex nature of our species. This book, for me, was a lesson on morality, psychology, and human nature.


One of the major issues underlying the science is that is often relies on scientists' personal integrity; that is, doing the right thing at the potential cost of funding, a promotion, reputation, the betterment of the literature, etc. Even the best and the brightest aren't immune to the pressures of competition. And, this is a slippery slope. You are rewarded for being first or for "finding" some type of result. This can very easily dissuade a scientist from the process of actually getting it right (or, hell, even getting it wrong and being ok with that). We learn by observation; it's one of the first lessons you learn in Psych 101. So, when you look around, and everyone else is skirting the rules and it's a sink or swim situation, you kinda just say screw it, I need to swim.


So, how we do fix this? Is it on the individual level of the human to be more moral? Or, does it require a paradigm shift in the culture of science? Perhaps both. Further, this is a chicken or egg situation. Which needs to come first?

Comment URL copied!
Margaret Landry
November 28th, 2019 at 3:37 pm

-Libby Landry

Comment URL copied!
Steve Haydock
November 28th, 2019 at 3:13 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

What a phenomenal opportunity we have with this book club. Just taking a moment to read the thoughts and opinions of my peers this morning has filled me with a profound sense of positivity. A renewed and reinforced sense of community.


My own thoughts on the book mirror a lot of what has already been expressed so expertly by those before me. A wide range of emotions from the extreme of utter frustration to hopeful optimism in the knowledge that, regardless of the errors and manipulations, we still hold in our grasps the ability to make the difference with our own message.


The evidence presented appears overwhelming and damning upon initial reflection. But as many of you have stated we operate in an imperfect world. Our goal, our mission, remains constant. To strive to be better, to do better for those in our care. To not underestimate the role that we play in spreading the message that we are privileged to deliver.


To all of you I charge you to continue your efforts. Speak the truth. Do it truly in the knowledge that what you speak is the truth and is a truth that must be heard.

Comment URL copied!
Matt Lodin
November 28th, 2019 at 4:29 pm

A renewed and reinforced sense of community sums up my thoughts exactly.

Comment URL copied!
Natosha Haydock
November 28th, 2019 at 3:03 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Honestly a tough read for me. Not because it wasn’t well written or insightful but because it’s disheartening to once again hear that money and the desire to be acknowledged outweighs truth and the responsibility to give that truth to the public. Doctoring data and publishing false information because of greed and pride, makes me feel helpless to right these wrongs. This book brings to light that dishonesty but also challenges us to find that truth and share it. So in the spirit of stepping out of my reading comfort zone, the positive from this read is that I can greater appreciate the honesty that exist in the science of Crossfit. Bringing light to past mistakes in Biomedical sciences and driving for change with what happens now. Be better in my roles and my commitment as a trainer and a voice for the Crossfit methodology.

Comment URL copied!
Victor Morris
November 28th, 2019 at 2:34 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

We have been hearing from Coach Glassman about “The Mess” for years now. Rigor Mortis reading was for me a more in-depth look at the situation we already knew was disrupted.


You can get really demotivated reading “how sloppy science create worthless cures, crushes hope and wastes billions” getting to know points where we were misguided in our goal to understand a human-biology related phenomenon or condition.


Sometimes is due to ignorance, sometimes another interests are in play. The fact - as I see - is that everything brought us to this point and we should be GRATEFUL for that. That was needed to get us here.


We can't go back in time. We can act now.


I don't feel I'm in the position of finding a solution to this situation but I do believe it is important to be AWARE of the situation though - and bring awareness also.


As Coach Glassman says, we have an elegant solution for the most vexing problem of the world - and that is CrossFit!


CrossFit in its political action of telling the truth and also in our practical application in our communities teaching people how to squat and eat - here is something I feel strong to contribute!


The problem as I see is that human race is trying to "figure out” Nature instead of observing and learning from it. The chronic diseases are the consequences and not the causes of why the whole system is sick.


Great book to bring discussions and to show us we shouldn't rely solely on others - neither on studies - to find out what is right for us.


Excited about seeing this process happening in our main site CrossFit.com.


Thank you, Coach Glassman. Thank you CrossFit.














Comment URL copied!
Matteo Pozzati
November 28th, 2019 at 2:33 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It was a very difficult book for me, I struggled to understand the details because biomedical science is not my daily subject. I found it still usable even for the novice readers since the main concepts and examples are repeated several times, especially in the first chapters, to make them obvious to the reader. This is a good point for Harris, because it makes a seemingly niche topic, as the evaluation of scientific research, a subject within the reach of almost everyone.


Has it shocked me? No, not at all. Harris confirmed what I have long believed: human beings do not easily escape hypocrisy. It is a transversal concept that pervades every field and action, whether voluntary or unconscious, and therefore does not spare the field of research. I mean hypocrisy as the absolute meaning of the term, as the simulation of a virtue or an intention that in reality is not there. It does not surprise me to see the evidence of corrupt studies, spoiled and conveyed with the intention of confirming a packaged or commissioned hypothesis, rather than divulging the truth. Who would do his best to prove the reliability of a theory opposite to that in which he MUST believe?

How is it possible that the use of wrong cells, the incorrect analysis of the results, the lack of replicability of the experiments and the guided selection of the results are the daily standard? It is obvious to me that I am not a man of science, how can it not be for scientists?


As Harris explains, unfortunately it is not the search for truth that moves the system. The whole machine is driven by the urgency of approval by the canonical institutions, by the frantic need to produce quantities, by the incessant race to the next budget and by the affirmation of one's hypotheses. A modus operandi that did not spare even CrossFit, when the NSCA commissioned the famous and false study on the percentage of injuries in relation to CrossFit training.


Is there hope? Apparently yes. Harris proposes alternatives to resolve this crisis of information through a rigorous methodology, which involves total disclosure, complete transparency, complete sharing and the necessary reproducibility.


However, I believe that research has brought us to where we are today and that there is always something to use in order to take a step forward in the right direction. A revolution in the research system that points to truth and transparency is certainly necessary, because as Harris has written "being right should matter most of all".

Comment URL copied!
Elie Margerin
November 28th, 2019 at 12:55 pm
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

RIGOR MORTIS: Comment une science négligente crée des remèdes sans intérêt, détruit des espoirs et gaspille des millions de dollars et d’euros.


Depuis plusieurs années, le Coach Greg Glassman parcourt le monde pour exposer à qui veut l’entendre comment, face au fléau des maladies chroniques, sa prescription en matière de nutrition et la pratique de la méthode CrossFIt® offre une solution élégante voir même optimale.
La lutte contre ces maladies chroniques, responsables de 86% des décès en France chaque année, est aujourd’hui son cheval de bataille.
Cette bataille se mène à bien des égards comme celle qui a permis d’établir de manière incontestée l’efficacité de notre programme d’entraînement. En procédant avec rigueur, exigence, en obtenant des preuves de cette efficacité, tout en inscrivant en faux d’autres systèmes moins rigoureux voire frauduleux.


C’est à l’épreuve d’un même examen que Richard Harris soumet le monde de la recherche médicale moderne pour en dresser un portrait peu flatteur.

Dans cette ouvrage, l’auteur nous plonge dans un état des lieux du monde de la recherche. Il part d’un constat simple: la grande majorité des expériences scientifiques publiées dans les grandes revues spécialisées sont pour la plupart non reproductibles, c’est à dire inexactes.


De ce premier constat découle un effet domino, si une première expérience s’avère inexacte, toutes les recherches basées sur les conclusions de cette dernière se voient d’office invalidés, des mois de travail et des sommes d’argent considérables employées inutilement.


A la lecture de l’ouvrage vous verrez que le monde de la recherche et la communauté scientifique semblent embarqués dans une inertie, incitant à publier toujours plus, toujours plus vite, à obtenir des fonds pour financer de nouvelles recherches, quitte à sacrifier la rigueur et l’esprit didactique sur l’autel de l’urgence et du mercantilisme.


Si vous êtes néophyte comme moi, vous comprendrez certains tenant et les aboutissant d’un système amenant à ces dérives. Vous en apprendrez davantage sur l’expérimentation animale, l’expérimentation sur les cellules cancéreuses et les traitements expérimentaux. Vous comprendrez pourquoi, en l’état actuel des choses, le « off the carbs, off the couch » est le seul espoir viable dans la lutte contre les maladies chroniques.

Comment URL copied!
Luc Millier
November 28th, 2019 at 3:07 pm

merci pour la traduction :)

Comment URL copied!
Matthieu Dubreucq
December 2nd, 2019 at 12:53 am

Je pense que tous devrais lire ce livre qui remet en question notre fonctionnement de la science moderne. Le fait que la société privée a la main mise dans ce qui se passe dans nos hôpitaux et dans ce que les médecins lisent. C’est très spécial, car nos pays ont une société d’état qui est en charge de notre médecine et on s’attend donc à ce que ce soit neutre et impartial. On ferme trop facilement les yeux sur ce qui se passe. Ce n’est clairement pas seulement un problème Américain et pour la moment la seule piste de solution semble l’éducation de la population qui pourra ainsi questionner et avoir un sens critique sur ce qui se passe. Faites parti de la solution!

Comment URL copied!
Jamie Johnson
November 28th, 2019 at 11:43 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Richard Morris' compilation of inept, fraudulent and neurotic practices that seem to plague biomedical research left me disillusioned. I wasn't sure what the real definition of what a 'scientist' or 'science' really was anymore.


The oxford dictionary defines a scientist as "someone who studies one or more of the natural sciences (physics, chemistry or biology). This definition gave me little insight as to what science or any scientists responsibility to humanity really is. Do they even have a responsibility to us? By the billions of dollars that is spent by governments around the world, a layman such as myself would probably say, 'yes'.


So I asked the next reliable source I would use outside of the Oxford dictionary. My kids.


Their answers to me was unanimous, "a problem solver".


When I think back to how science is introduced to us at school, they were precise with their prescription

.

Right again (as they are most days from their own point of view).


As youngsters we see science as a tool to gather evidence to solve the unknown. It's principle in theory is noble.


I Loved their definition so I asked more questions.....


What do you do if an experiment you perform confirms your aim?


They answer almost in sync, "we do it three times and if each experiment gives the same result, then we would say the experiment is a success".


I get it. When a science student can prove their theories three times with experiments it's gospel truth.


However, when I asked what they would conclude if the experiment proved your theory two times out of three experiments? Would you still believe in your theory?


Their answer was a firm "no". I asked why they felt this way and what would they do if it happened?


They proceeded to tell me how they would check their methods and perform the experiment again with the aim also focusing on how they may have messed up the experiment that didn't match up.


The integrity of their teenage innocence lead me to ask... "what age is it that money and ambition could turn a bio medical scientist into a liar.


Poor lab technique, the inconsistency of mice DNA and misdirected trust of previous clinical studies are almost forgivable in comparison to the cultural norms that sit deeply within the Biomedical research core. To cut corners, to lie and mislead is as cruel to mankind as any genocide that we have endured in our short time on this Earth. We trust these humans to find cures for Cancer.


Just as alarming after having read this book a few times is that most (not all) of the billions of research dollars that are being wasted are directed towards curing chronic disease. Diseases that we are conquering in our gyms each day. With zero government funding.

Imagine a world where the money spent on chronic disease cures were spent on educating man kind about the benefits of eating meat and vegetables, nuts and seeds, some fruit, little starch and no sugar. Then the few shining lights in our scientific world like Glen Bagley could be given the responsibility and funds to cure the unknown tragedies that are ALS and glioblastoma, to name just two.


Apart from this I offer no solution. Other than potentially putting our teenage children in charge of biomedical research.


Before they can tempted by the ugliness that money, fame and peer acknowledgement has bought to this "broken" fraternity.

Comment URL copied!
Oliver Wilson
November 28th, 2019 at 11:31 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

A recent survey by the Australian Broadcasting Corporation indicated that scientists are the second-most trusted profession, trailing only to doctors and nurses. In his book Rigor Mortis, Richard Harris illustrates the countless billions of dollars wasted, and the resulting countless lives impacted by the challenges of conducting sound research. Being a practitioner of the scientific method is fraught with complexities – the book divulges a broad number of error ranging from the height of the shelves animal subjects are placed on, through to accidental contamination or incorrect labelling of samples. Although there are arguably innocent mishaps that occur, the effect they have can cause exponential damage. Beyond unintended mistakes, the book reveals the pressures researchers are under – the push to publish in pursuit of prestige, funding (motivated by both best intentions and more sinister agendas), and the shift in scientific culture to impose the expectation to discover results. These have resulted in a reproducibility crisis, where but a fraction of studies can be replicated to indicate findings consistent with peer-reviewed papers.

Peer review in-and-of-itself contains challenges – as stated in his book Peak, Anders Ericsson (author of the research that uncovered the ’10,000 hours to become a master’, popularised by Malcolm Gladwell’s Outliers – which itself is not as straightforward as it is presented) illustrates that panels of experts select the experts that sit on the panel, effectively creating an environment ripe for groupthink. Further, this dissuades exposure of poor science. As stated in the book, ‘the people you criticise might be reviewing your grants’.

The marketability of scientific research, due to the unimaginable profits that can be made from breakthrough medications and treatments, has put the entire industry into a pressure-cooker – survivors are selected by their ability to uncover significant results. We must view research through a lens that appreciates not finding anything – this is still a result, as it narrows the scope and introduces a process of elimination. With careers, funding, and lives at stake, the scientific world has elected to prioritise breakthrough science over rigorous practice.

Comment URL copied!
Weon-Woo Lee
November 28th, 2019 at 11:21 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This book tells us the uncomfortable truth of nature. Everything has its weaknesses. Science is the same. Scientists try to get close as possible to the answer. And they are the group of people who can stay closest to the scientific truth. However, they and their researches are not perfect, and there are lots of rabbit holes just like this book says. We need to be aware of this and help scientists to work as hard as they can to do their job right. We need science but should know it cannot always be right or truth. That means we need to use it and get help from it, but we cannot rely on those only. We need to be more proactive and have ownership of our health and life. By constant educating ourselves and knowing how to care about our health will help us to filter out the wrong science and help the scientists who are trying to do the right.

CrossFit coaches teach their clients how to control their life and health. We teach how to track the health markers, performance markers, and ways to find the right food and proper quantity of food for each individual. Simply, way to own our lives. It’s great to know CrossFit coaches are heading in the right direction.

Comment URL copied!
Tiago Oliveira
November 28th, 2019 at 3:36 pm

Completely agree with your point on ownership. We need to keep doing what we are doing with CrossFit, health and fitness and use science as a guide. not as the absolute indisputable truth.

Comment URL copied!
Lachlan Learg
November 28th, 2019 at 10:33 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Unlike most others I actually wasn't surprised by what I read in 'Rigor Mortis'. I'm fortunate enough to have several medical professionals as members of my affiliate and we engage in discussions and readings of such topics regularly. So, much of what was raised in 'Rigor Mortis' wasn't a revelation.


It's easy to become jaded or even somewhat nihilistic about 'the mess' and to have numbers put to many of our cynicisms is down right depressing. John Ioannidis at Stanford for instance reviewing the vast sea of genomic papers to discover that only "1.2% of studies stood the test of time as truly positive results" in that particular area of research. To think that merely little more than one percent of studies into genetic links for obesity, depression, osteoporosis, high blood pressure, etc. have survived any rigor doesn't exactly instill much hope in the advancement of scientific research and hence medicine. The projection that at the current rate all new medical advancements will stop by 2040 speaks to the stagnation of an entire discipline. And really shows that the pessimism that Glassman has expressed doesn't at all seem to be an overreaction when spending any time reading into it.


However I'm still optimistic for two reasons:


One, I'm hopeful that advancements in quantum computing could then carry over into the broader scientific and medical fields. Speaking to one particular member at my affiliate who is a pharmacist about 'Rigor Mortis' he was unfazed by what I was telling him from reading the book. He knew that pharmaceuticals were a big business and he knew how that business operated. He wasn't surprised by the fact that ultimately humans are being humans. The faults in biomedical research or commercial pharamaceuticals are a reflection of humankind's shortcomings and philisophically one could argue they will always be there. We as an organism are driven by personal success and often greed. So to that end we can escape this loop and be set free when computers can do the leg work for us.


If quantum computing gives us the ability to perform years worth of calculations in seconds and can sift through unimaginably large fields of data just as fast, it can be an invaluable asset in the pursuit of better science. Rather than being limited by our all too human faults that are highlighted in 'Rigor Mortis' scientists could be aided by an impartial technology that shows the truth not what we fool ourselves into seeing.


This pharmacist's belief in quantum computing being able to offer us not only better, cheaper, faster pharmaceutical development but also better scientific advancement made sense.


But rather than being jaded which we both could be "knowing too much" so to speak; as we discussed the book, 'the mess' and quantum computing I was reminded of the other reason I'm optimistic.


I'm optimistic because I can have discussions like this with members in a CrossFit affiliate. There are people out there in the medical and scientific world who identify with 'constantly varied functional movement at high intensity' and attach their knowledge to what we teach and together we all grow.

I'm optimistic because they know things about exercise, physiology and health now that they didn't know before stepping into the world of CrossFit.

I'm optimistic because their scientifically geared minds are open to new information and are excited, not threatened by what we have to offer.

I'm optimistic because we can engage in discussion about how horrible the double unders were today as well as biomedical research, all in the four walls of an affiliate. A place that is designed for exercise but also community building, thinking and so much more. I think that's pretty special. And I think so long as there's CrossFit affiliates and people from all walks of life including scientists and medical professionals in those affiliates we have a chance to change the world together.

Comment URL copied!
Joe Pearson
November 28th, 2019 at 10:07 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I really enjoyed the book, and believe there are take away lessons that can be applied to any profession or discipline.


One of the big things that stood out for me was errors in previous research, were compounded when old studies data was used for meta-analysis. This mirrors the 'novice curse' that we have all seen with ourselves and in our members. The rush to get to, or focus on the high skill, 'sexy' movements at the expense of mastering basic movement patterns. If you do not have a rock solid air squat, the faults you have will be magnified as you move to the front squat, and the overhead squat.


I understand the pressure of having to be at the 'cutting edge' of science, and to come up with something new in order to be relevant (and financially supported) in your field. However, there seems to be major value to the wider population in concrete, repeatable experimentation on the basics of health, training and nutrition.

Comment URL copied!
Natalia Diez
November 28th, 2019 at 7:39 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I thought Harris did a great job spelling out all the problems in the field. Biomedical science is hard for me (general audience) to understand and Harris did a great job with a clean and easy reading.


Some of the information did not surprise me but the fact that many scientists are fully aware of the problems and go along with the system is disturbing.


Harris provides numerous examples of specific papers that are incorrect and examples of studies that evaluate the problems in scientific literature. The problems are related to misidentified cell cultures, poor study design, poor data analysis, bias, selective reporting of study results, and sloppiness. Some of the studies still provide useful information, but others lead to dangerous conclusions.


The author discusses several practices that may help the crisis in scientific research and publication. Harris explain potential improvements in they way errors are detected and disclosed. We need an improved education in research methodology and statistics, we need to share data (open data) to improve scientific rigor. This open data would provide accountability and allow other scientists to evaluate the strength of the studies.


Harris’ book isn’t just a description of the state of the field, he provides concrete adjustments and changes that can be made to improve the quality of research being done. My only complaint is that I wish he had gone more into making the case for why we should still fund science.


At this point my believe is that Science needs to be reinvented.


While reading this books, (maybe because I am a trainer) I found several similarities with the Fitness world, food and beverage industries. A system with lies, uncovered data or selective reporting of study results. Lucky enough, we have CrossFit ;)

Comment URL copied!
Natalia Diez
November 28th, 2019 at 11:21 am

At this point my believe is that Science needs to be reinvented.


While reading this books, (maybe because I am a trainer) I found several similarities with the Fitness world, food and beverage industries. A system with lies, uncovered data or selective reporting of study results. Lucky enough, we have CrossFit ;)

Comment URL copied!
Zachary Pine
November 28th, 2019 at 7:36 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Start of a beautiful thing. Thanks Coach!


What is amazing about humans is that the truth is so intriguing we have to pursue it at all cost. On the other side we hide the truth at all cost for personal gain. I have 2 daughters that use both of these methods everyday. If you have worked with kids or are a parent i am sure you have witnessed the creativity of your little ones under the stress of truth. Finding the truth in science is a huge task with how we evaluate what is TRUE (statistical significance) and not. I usually pin them down to the ground and make them tell the truth, but the big medical companies seems to have a little more body weight in the game. If the theory is true we should be able to reproduce that multiple times over not slide by with a study that is designed to be statistically significant. It is like telling little lies to get out of being pined to the ground.


Science drives inspiration and I will leave the pursuit of “rigor” to the scientists, you know… the ones with lab coats and petrie dishes, but we all can be inspired by their efforts. Will the scientific theory become the scientific truth, humm.. where technology and science advance so far that it becomes what we look at before we actually look at what is happening in front of us? Probably not in my life time, but the ideas inspired by science drives our human race. Makes us look at situations in different ways and push the boundaries of possibilities.


Rigor or not science will be evaluated through measurable empirical data. The evaluation process of the validity of all experiments needs to be assessed, but if there is no experimentation we loose inspiration.


Every time I coach is an experiment!

Comment URL copied!
Victor Morris
November 28th, 2019 at 1:05 pm

Simple and strong point, Zach!

Comment URL copied!
Emily Jenkins
November 28th, 2019 at 7:25 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis by Richard Harris puts biomedical investigations under the microscope in order to point out some highly problematic practices. From flawed raw material to skewing data, Harris outlines some shockingly ordinary issues in the field. One of the root causes for these costly mishaps has to do with academia’s premium on publishing. Pressure to publish in terms of quantity vs. quality regularly encourages scientists to put forward cases that are flawed at best and at worst, untrue. The astonishing absence of scientific rigor, from which the book gets its name, is directly correlated peer-reviewed publishing sources. I would like to put forth my two cents concerning the publishing industry:


Just because something is published, does not make it true. Before the World Wide Web in 1993, it seems that published material was more “reliable” because those with access to the media tended to be professionals. However, professionals are not exempt from cheating and deceiving, this includes medical professionals despite society’s seemingly inherent respect for them.


Peer-reviewed journals are often working with an insanely small budget, especially in comparison to the spending involved in the studies they publish. Furthermore, most of those involved in the publication processes have other full-time jobs. They, by nature, have other priorities.


For some fields, publishing on a “failed” hypothesis does little to advance the discipline. I would argue that that is not true for biomedicine. If one experiment is run well, and the results are unable to prove that the theory cannot be incorrect, that work could still be valuable. Maybe another scientist at another lab was about to perform a very similar study and now, can shift one factor to see if that has a different effect. Furthermore, if the results of the “failed” experiment are published, other scientists can contribute to ideas for ways to modify future studies in order to work towards a more meaningful investigation. If the goal is to advance the discipline, I think that publishing should happen often. However, we have seen that without publishing the negative has resulted in an overproduction of the sloppy. This system has to change. Harris talks about certain badge systems in place for particular journals that highlight respectable investigators, however, there are still false studies being cited on a regular basis.


Solving issues associated with “Rigor Mortis” in the field of biomedicine would involve taking on a host of different factors. The publishing industry, in my opinion, would be a simple way to start.

Comment URL copied!
Antonio Aguirre
November 28th, 2019 at 6:55 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Although I find Richard Harris' findings quite illustrative I do not think this information has been unknown by anyone who has been involved in scientific research or by regulatory institutions and policymakers.

Even though we should be comprehensive of the number of intrinsic difficulties biomedical research has to deal with in comparison with other scientific fields (most of which the author properly describes in the book), science, in general, has become a hypercompetitive environment, promoting scientists' careers by the number of their publications, rather than quality, resulting on that continuous trickle of bad science and useless effects.



With all that said, it causes me concern, readers may consider this book as evidence against science and the scientific method. As I see it, acquiring knowledge through an empiric method has brought our civilization to where we stand at the moment and it is precisely that TRUE science the one I think Richard Harris is reclaiming throughout his book.

Comment URL copied!
Tiago Oliveira
November 28th, 2019 at 6:53 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

The word mess has a whole other meaning after reading this. The system encourages deceit in a field where the ultimate goal is truth. Being a scientist is not easy and the variables that can go wrong are infinite but there’s simple stuff that could go a long way. Things like publishing not only experiments that seem to work but also the ones that fail so their peers don’t waste time and money on something similar. Accountability to previously published studies that have been proven wrong or misleading and so on.

In my humble opinion any system that focuses on results, like finding the next new thing all the time, will sooner or later fail.

We need to grant grants to those scientists that do deep responsible work and stop fueling this half assed machine gun of bullshit we call science.

Comment URL copied!
Kurtis Bowler
November 28th, 2019 at 6:15 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I was with Cherie on this in feeling angry and disappointed.


Not angry that they were wrong, but angry that we can't really even know if they were wrong in a lot of cases because they weren't even following the basic protocols to show the results were accurate. We have to have accurate results to learn anything.


Angry and disappointed that if they are wrong they can't or won't admit it. We have to be ok with being wrong if we want to learn anything or get better. In my personal experience I have learned far more from mistakes than getting something right.


Disappointed that most don't even seem to care or are too afraid to speak up about it.


The one thing it did was cement the fact that the best that we can do is make sure the people in our care (family, friends, affiliate members, and seminar participants) know the difference. If we keep doing this it will change on a larger scale. Look how far we have already come.

Comment URL copied!
Joe Masley
November 28th, 2019 at 12:52 pm

Love this Kurits. I think it's easy reading this book to try and identify strategic solutions to the entire field of biomedicine. When in actually, the most important thing we can do is make a difference in the lives of those in our care by educating them to these issues - thanks for this reminder.

Comment URL copied!
Eric O'Connor
November 28th, 2019 at 10:09 pm

Kurtis, I was also floored that some scientists can't or won't admit it when they are wrong and, like you, I have learned much more from mistakes vs getting things right. At one point I was a little sympathetic to some of these researchers who have spent an entire career on a single experiment, trying to do a good thing, only to be proven wrong. This had to be hard for them to digest, but in the end, you need to leave the ego at the door and admit the failing and/or others need to speak up! Extremely unfortunate that ego has gotten in the way of the truth at times.

Totally agree with you that a big takeaway for me is to make sure that I'm taking care of the people around me, educating them, and helping them to not be misled.

Comment URL copied!
Richard Gonzalez
November 28th, 2019 at 6:14 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Reading Rigor Mortis has opened my eyes to the rampant ills of biomedical research. Rigor Mortis reads like an exposé, with the shock value of a Rolling Stones article, but the sterile objective view of NPR investigative journalism. There is a problem in the scientific community, with institutions that covet research dollars, and the strain of scientists to publish first. Novel ideas get research dollars. Corner cutting and p-hacking are inevitable in the hyper-competitive scientific research environment. As the title of the book implies, there is a systemic lack of scientific rigor and in some cases, fraud.


Although the disappointing, the book is not a depressing, but a grim wake-up call. It is a call for academic institutions to initiate graduate problems with a deeper understanding of statistics at the onset of student programs, so they can learn to read, understand and dissect the research. It is a call for academic institutions to engage and promote reproduction studies as important, maybe even more-so then-novel research. It is a call for academic journals and scientists to publish comprehensive methods, data​ and results so others can reproduce and find flaws in the studies. It is a call for organizations to ignore the impact score and popularity of journals in which scientists publish.


I have learned a little, but it is my responsibility to learn more. I will support those that are promoting scientific rigor and try to understand this “mess” a little bit more each and every day.

Comment URL copied!
Matthew Swift
November 28th, 2019 at 6:05 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

To say that I found this book alarming would be a gross understatement. Coach has talked about “the Mess” that is the health industry. I remember hearing him say that he holds no hope for the mess being fixed. At the time I thought he was being pessimistic, but after reading this book, it is hard to see how the described problems of biomedical research could be fixed, let alone an entire industry that is built upon the flawed product of that research .


Richard Harris states “bad incentives and bad habits are both to blame”. How do you change the incentives and habits of an establishment?


My understanding of the problem, as described by the author is:


- The scientific method is meant to prevent the scientist from being fooled


- Lack of rigor in applying the scientific method leads to false positives


- The false positives could be detected and avoided if scientists systematically attempted to reproduce results


- But seeking reproducibility requires a high level of rigor, which is the thing lacking in the first place


- Once research is published it is not likely to be challenged or changed, because the publishers have indirect power over scientists’ careers


- If published research is challenged, and can’t be reproduced, lack of reproducibility is often ignored, ironically as a false negative


- False positives continue to be used as a foundation for development of drugs and treatments, even once detected, due to a very low level of retraction


- Many people may be acting in good faith, but fooled by the research they are basing their work on


- Powerful commercial and career incentives exist to publish research, this leads to hypotheses being changes to suit the data after the fact


- The are strong disincentives to challenge research and published papers


- Fundamentally, once enough people have benefited financially from the false positive it crosses into the realm of “accepted truth” and there is too much at stake to acknowledge that it is wrong


It would be easy to feel lost in this mess. I kept reminding myself that the very fact that this book was written is proof in itself that there are still many within the system trying to fix it, and I take comfort in that.


There were three thoughts that this book prompted in terms of coaching and the affiliate.


Firstly, doing things correctly (rigor, excellence, virtuosity) requires effort beyond immediate reward. It is hard to hold yourself to a higher standard. Although it is worth it, but don’t expect the industry to reward you. “Rising above” will require some personal sacrifice. Rigor requires effort and there is generally no incentive other than a personal one. Perhaps that is why exceptional people gravitate to CrossFit in the first place, because virtuosity is culturally rewarded.


Secondly, there are often commercial rewards for cutting corners. Harris noted “labs that use quick-and-dirty practices will propagate more quickly than careful labs”. I can easily find the analogy in the fitness industry. The remedy to this is not cutting corners, but rather setting a standard that makes quality evident to the consumer. Given a choice the scientist will choose a better lab. We need to continue to make sure the world knows our "labs" are better.


Finally, in seeking truth, we walk a fine line of knowing when to ignore criticism and when to accept it. It is clear from the many examples in the book that the inability or unwillingness to accept criticism prevents the ability to correct errors. However, to bring about a change, there is also a requirement to ignore the criticism of the establishment. No doubt, Harris has been heavily criticised for his work. Being too sensitive to criticism keeps us within the status quo of the consensus, but too little sensitivity to criticism makes us dogmatic in our beliefs. Therein lies the challenge.


For me, the overall takeaway was ... to be more right you need to be willing to be wrong, and everything about that works against human nature and normal incentives. What a valuable book!

Comment URL copied!
Wes Piatt
November 28th, 2019 at 5:54 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

“This was from a peer reviewed study”. “This was double blind placebo tested”. “According to scientists at (place whatever high level intellectual academy you want here)”.


Raise your hand if you’re guilty of hearing these phrases muttered after a point is made and regardless of wether you believed what you were hearing was true or false, all of a sudden you found yourself saying, “Well shit... I guess it must be true”. 🤚🏼


It wasn’t until last year when I heard Coach Glassman speaking to a room full of L1 participants at the Ranch in Aromas, California that I started to look into “bad science”. A participant asked his opinion on a peer reviewed study that was put out about hydration. Glassman kindly went over his opinion on the subject and toward the end of his response he mentioned the words “peer reviewed” with a chuckle, as if the words didn’t move his opinion in any way!


This surprised me and immediately had me scouring the internet to find reasons why I should no longer blindly agree to “scientific research” or “peer reviewed” as true. Instead of using these studies as facts, I now know to use them to ask more questions and hunt for the truth. Richard Harris agrees and reports the problem with science after 1980 is that nobody is checking to see if this research is really true anymore.


As I dug through Rigor Mortis, I found myself very proud to be a part of CrossFit. CrossFit as movement has questioned 3 X 12 - 15 and that steady start cardio on ellipticals was the path to heath and fitness and instead said “show me the data!”.


Scientific data needs to be observable, repeatable data - with an emphasis on REPEATABLE. This point is what stuck out most to me in Rigor Mortis. We want to have a set of checks and balances on the basics (points of performance for us coaches) when it comes to testing. We want to ask “how” or “why” no matter how obvious it seems. We want to look under the covers, and peel back the onion to find what’s hiding in plain sight.


It’s this constant pursuit of virtuosity that science is currently missing, but my hopes are high! Seeing what CrossFit is doing with CrossFit Health and pushing to get books like this out there to the public is slowly moving the compass in the right direction. I’m happy to have my daughters growing up in a CrossFit gym, around positive influencers, and truly knowing what they need to do in order to live a long and healthy life.

Comment URL copied!
Eric O'Connor
November 28th, 2019 at 9:55 pm

Wes! I have totally been guilty of believing a study if it was labeled as "peer reviewed", "according to scientists" etc. Heck, in college, those were the exact things that my professors told us to look for in any study! I'm glad this book shed some light on strategies to analyze studies to assess for their validity. Like you, the point that stuck out to me the most was the lack of repeatable of successful studies. This is something I had not considered much before, but the data was extremely eye-opening.

Comment URL copied!
Drake Sladky
November 28th, 2019 at 5:07 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

As I read through Rigor Mortis our own Charter became a repeated thought in my head. M-C-I does not just apply to how we conduct business but should also be how it is conducted in the world of Biomedical research:

Mechanics - How can there not be standardized testing procedures across the industry? Without such standards how can we expect there to be any consistent reproducibility across testing? By applying and enforcing some simple testing procedures and making sure the results reflect what was initially stated were being tested for, mechanics can become the foundation which research is based on in order to ensure reproducibility and credibility.


Consistency - Possibly due to this lack of standard testing procedures but also due to ones ability to self interpret/manipulate results, we do not have the consistency/credibility within the industry. One cannot build upon someone else’s findings because we do not know if they are accurate.


Intensity - The author notes at the end that more is just more and does not mean it is better. People are getting ahead in the biomedical research field by publishing a lot of information not by publish quality information. There should be a balance, yes we do need more research but at what cost. Can the research field be pushed forward by the self application of Threshold Training,? Can we slow these people down a bit so their quality of work improves and then once we have done so maybe they can forge ahead at a speed that is maintainable while minimizing errors within their research?


Just my thoughts?

Comment URL copied!
Emily Jenkins
November 28th, 2019 at 7:22 am

Nice analogy Drake, I agree.

Comment URL copied!
Matt Lodin
November 28th, 2019 at 4:33 pm

M-C-I rang true for me as well,


Except it was morals, character, and integrity. All which had a theme of being lacking through this book.

Comment URL copied!
John Brown
November 28th, 2019 at 4:54 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Rigor Mortis was an interesting read that points out some interesting, but not necessarily unsurprising issues in the field of Biomedical research. The interesting part to me is that it forgets that humans are flawed by the very nature of being human but expects us to hold these particular humans to some mythological standard that is idealist and unreachable. The unsurprising part is that it offers no solutions to go along with it's accusatory finger; why? Because the problems that it points out are infinitely nuanced and complex. It is not as if we can replace mouse models with human models. Hell, most of the scientific experiments that many of my friends in the research field would not even pass an ethical review board.

What Harris does do for us, the uneducated reader, is suggest that maybe we should not put our scientist or the process of science on a pedestal. To remember that they are in fact humans and thus succumb to the human condition. Does that need to improve? Absolutely, but that does not mean that in order to improve that we must subvert the entire history of science just because the methods at their disposal were less than optimal or that their moral compass was broken. What it also means is that we must continue to challenge science to improve, to hold themselves to a higher standard and to find funding solutions that do not reward shoddy work... How do we do this? I do not know. I teach people how to squat. What I hope is that science can start to police itself from the inside. While it learns how to do that, the best that the rest of us can do is to continue to hold ourselves to the highest standard that we can in all of our various walks of like in order to lead from the front.

Comment URL copied!
Richard Gonzalez
November 28th, 2019 at 6:25 am

Nice read, John. Arete - Excellence of any kind.

Comment URL copied!
Peter Shaw
November 28th, 2019 at 11:56 am

Well said, John. I agree.

Comment URL copied!
Joe Masley
November 28th, 2019 at 12:34 pm

Agreed, John. I think that self-policing may be one of the most needed solutions to fix these issues that exist. It falls on your point on human behavior - scientists have to want to uphold excellence first before any real progress can be made.

Comment URL copied!
Jennifer Hunter-Marshall
November 29th, 2019 at 6:55 am

Agreed.

Comment URL copied!
Patrick Barber
December 2nd, 2019 at 5:33 am

I love it John.

Comment URL copied!
Embriette Hyde
December 2nd, 2019 at 10:59 pm

As someone who has been on the inside, I couldn't agree more. Nothing is more demoralizing than having an adviser hark about the importance of accountability and doing things right, but then accepting without question the fact that one of his research papers was published WITHOUT GOING TO PEER REVIEW just because one of his good buddies was the editor of the journal. He didn't see a problem with it, why? Because he honest to God thought that he was correct and everyone else was incorrect, so the reviewers wouldn't have found a problem with his work anyway. Amazing. Sad.


There is an online pre-publication site that publishes scientific articles that haven't been published elsewhere yet. The site is called bioRxiv. While it doesn't use a peer review process, all articles have comments enabled, allowing the community to "peer review" any article they wish. I wish that this were used more by the scientific community.

Comment URL copied!
Kelly Brown
November 28th, 2019 at 4:32 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

This book was a difficult read for me on multiple levels, as trainer, a clinician, and an individual for which the battle with chronic disease is uncomfortably close to home. My initial response was the predictable horror that so much inaccuracy occurs, and worse, gets published. Laziness and pride are inexcusable in a game where the stakes are this high. It is one thing to miss a corner when sweeping the floor; but it is quite another when inattention to details is potentially costing people their lives.

However, I agree with others who have acknowledged that some of these errors were not to be reasonably anticipated. Some of them were straight up, "Wow, I did not see that coming" issues. When one is exploring the unknown, it is not reasonable to think that you will be right even half the time. It is upsetting to think that researchers are in position where they feel they have to contrive to show results that are not there just to keep their jobs. But I get it. When we try to monetize the distribution of "truth" it can get pretty slippery.


After reading this book, I reached out to a friend who is a longtime member of the research community because I was considering getting involved in a clinical trial myself, and frankly, after this read, I was dubious if it was even worth the risk. His response was thought-provoking. Just because the system is flawed does not mean it is useless. If everyone refuses to participate in clinical research out of fear or disdain, then we literally will learn nothing. Yes the accountability needs to improve. But I'll be honest, I have no idea how to accomplish that. The watchdog organizations Harris praises are doing a good job of pointing out the problem. Maybe I missed it, but I would have loved to see an actionable proposed solution.


In the meantime, we should be educated consumers. If an article references a study, we should take the time to look it up and see if it passes any reasonable validity/applicability standard before we quote it at the next dinner party, or worse, make recommendations to our clients.

Comment URL copied!
Chuck Carswell
November 28th, 2019 at 7:30 am

“Laziness and pride are inexcusable in a game where the stakes are this high. It is one thing to miss a corner when sweeping the floor; but it is quite another when inattention to details is potentially costing people their lives. “ Rigor Mortis sparked fluctuations between confusion, frustration, and downright anger. I’ve resolved my thoughts to: just as every athlete needs a coach, every researcher needs accountability, and every prescribed medicine needs data reflection for best solutions. Otherwise, the game continues with new contestants.

Comment URL copied!
Thiago Borges
November 28th, 2019 at 3:54 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

The book deals with an extremely important and pertinent subject for society. As it studies health, the book generates questions and frustrations with its statements. It is striking to realize that research has in some places become a market, where scientists end up publishing materials that have not always been 100% selected to continue having investments.

On the other hand, a humanization of these professionals by the author leads us to realize that the obstacles to the publication of a research are enormous and often leave the researcher dependent on several factors. Imagine the work of a lifetime: Now get people paying to refute it, adding journals that make a judicious selection (to some extent) and a need for investment. Also, of course, human factors like the disappointment of seeing that your work is not doing very well and are criticized for that. All of these problems lead, in my view, to wasted time and money on research that goes out of place or goes vague as a "fight of egos" continues to grow.

Another very relevant point that should be quite explained and discussed is the fear of failure. Culturally you cannot go wrong, you cannot give up, nor can you slow down the processes of your life, which leads to grown-ups who are afraid of being wrong and would rather waste time than make a mistake and risk other ways. But what is the relevance to our life of this thinking? We must learn to recognize that we make mistakes, accept corrections, and improve on everything we do. Or maybe even broaden our horizons and try new projects.

Comment URL copied!
Gene Williams
November 28th, 2019 at 3:22 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

The author, Richard Harris explains in this book how science goes wrong. There are so many ways for scientists to make errors in research, experiments, data collection and publishing that it’s a wonder they are ever successful. Sometimes errors are a result of sloppiness, other times a result of unforeseen and uncontrollable variables, and others a result of a publishing system that has incentivized fast and novel papers which end up being retracted.

Scientists are also up against the wall, being forced to publish papers in order to maintain their positions or being asked to show the VOLUME of their work to gain a position instead of the quality. With the research community set up like this it is unsurprising that mistakes are made, both unwittingly and, to lesser degree, wittingly.

I found his descriptions of the many issues facing researchers and his anecdotes about specific experiments gone wrong quite interesting. Who knew that tests run on a machine one day could yield different results if run on the same machine on a different day? Or that if the type of test tube used in one lab was different from another lab the entire experiment could yield different results? Or that the mice used in an experiment could have different temperaments depending on how close to a light source they are kept? There is a seemingly endless amount of ways an experiment can be affected. There is a movement to make much of this better with guidelines on the design of the experiments themselves, better interpretation of the data using statistics, and open sourcing the methods and materials used.

How I relate this to my daily practice as an affiliate owner and a member of the seminar staff is a bit of healthy bit of skepticism for the near daily barrage of headlines about “scientists say xxx is linked to yyy in new study” that either my clients will see or ask me about. Knowing about some of the pitfalls that researchers risk falling into will help me read the study more carefully, see where the issues could be, and make a more informed opinion about the stated conclusion. Many times I’ve spoken to a member about nutrition and they say “but I saw on the news that _________ is bad for me.” This gives me a little more ammunition to debunk some of these claims.

The good thing is that, eventually, the truth emerges. It may be slow, and it may be difficult to change ideas or habits when the truth is discovered, but science does correct itself. My hope is that the time and dollars spent are used more carefully so that the medical research becomes more efficient.

Comment URL copied!
Embriette Hyde
December 2nd, 2019 at 11:22 pm

"Scientists are also up against the wall, being forced to publish papers in order to maintain their positions or being asked to show the VOLUME of their work to gain a position instead of the quality. With the research community set up like this it is unsurprising that mistakes are made, both unwittingly and, to lesser degree, wittingly."


You have hit the nail on the head. There is a heartbreaking story that happened not too long ago in which a graduate student sadly took his life because he and his adviser could not see eye to eye. Long story short, the student wanted to uphold the rigor of scientific research that we, as graduate students, all start out believing in and believing everyone else does. After several threats from his adviser, this poor student unfortunately felt the only way out was to take his life.


This is just one example, but scientists at all stages of their careers experience depression (at an alarming rate) and all too many choose suicide as the solution. We don't WANT to contribute to the problem of irreproducible science. We do what we can with the money we get. We try to keep our careers because it's not just our families that depend on us, but the families of the students and post-docs in our labs. When we are pushed up against the wall, bad decisions are made. Sometimes that means sloppy science is published. Sometimes that means lives are lost.


We desperately need to change the culture of academic research.

Comment URL copied!
Joe DeGain
November 28th, 2019 at 3:16 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

“How sloppy science creates worthless cures, crushes hope, and wastes billions.”

This quote from the front cover is something we all believe (I think). The eyebrow-raising facts that are presented in this book expose that MOST science IS SLOPPY science. The question is, how do we stay away from this mess? How can we self-advocate for ourselves so we do not have to be dependent on the crooked sciences? If we could only find a lifestyle that would keep us from chronic disease… then we would not have to be so dependent on the sciences… MAYBE that is what we should aim for. A lifestyle that can mitigate the chances of chronic disease, mitigating the chances of us needing to be dependent on the sciences. If you have a potential solution, feel free to list it in the comments below.

I suppose we should show up to life, every day. When people ask why/how you look so healthy, tell them what you do. BE THE FRONT LINE.

Worthless cures are at an all time high. Mostly because the knowledge base used to create the cures is based on sloppy lab work performed by previous researchers. We can also couple in that the preferable results of lab work is fabricated by big Pharma or other money-hungry agencies in order to sell more drugs. In addition, the genetic differences between humans and lab rodents combined with the environmental temperament of even the best lab rodents adds to the complexity of the science.

I believe at one point the book relays that by the year 2040 the volume of sloppy/fabricated science being used as a knowledge base for drug research will be so high that science may just come to a screeching halt. Well, that is just crushing hopes all around. Add that to the tangible evidence of poor science/drugs literally keeping people sick (and even killing people) in order to make money and we have even more reason to lose hope.

It logically makes sense that this practice is wasting billions. I mean, if grants of our tax money are being given to scientists with a poor knowledge base, and big pharma is paying scientists to fabricate results, I can’t begin to wrap my brain around the price tag.

However, there are experts coming forward and exposing the truth. This suggests that we may be going in the right direction. There are some politicians going to bat to help stop the bleeding. This is more evidence that there is hope. There are, occasionally, benefits that come out of the poor science (I think Rogain was mentioned? I may need that soon!). THAT is hope! There also is scientists who are fed up with their field, who can hopefully be enough to turn this field around. THAT is hope. Lastly, there is the front line. I guess that would be anyone expressing a lifestyle that could mitigate the need to depend on the sciences… If we throw that in, maybe we could shift the dogma beliefs in society today.

Comment URL copied!
John Brown
November 28th, 2019 at 4:36 am

Joe,

While I agree that sloppy science costs billions of dollars and a lot of pain, to say that "most science is sloppy science" is a bridge that I am unwilling to cross. Sloppy compared to what? How much scientific research exists that we are unaware of? How much of it is cutting edge and thus expected to have some significantly greater tolerance for error? In some cases the scientists that we speak of as sloppy are literally creating the math needed for their research as they go. At what point do we call that learning as opposed to sloppy?

Comment URL copied!
Dave Eubanks
November 28th, 2019 at 3:03 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

Reading through this book I, like Cherie, found it disheartening at first as Harris outlined in detail the seemingly innumerable ways in which the presumably well-meaning scientists in various fields churn out either worthless or more of greater concern, harmful and wasteful studies.


As I continued though, I found myself looking for whatever beacons of hope I could find in the text. I was trying to find something to make me feel like things could get better or the tide could turn. Geneticist Ron Davis of Stanford supplied a beacon when he said: "the power of science is, the truth will eventually come through."


These beacons became the experts in the field with enough knowledge to be able to ferret out the issues with papers claiming novel results. Experts like Christopher Korch at the University of Colorado when he was able to recognize the issue with a seemingly "immortal" cell line produced by folks at Ohio State. Unfortunately for us laypeople, these issues aren't as readily apparent. Though it did remind me of a study I read in an NSCA publication shortly after becoming a personal trainer which touted the efficacy of the Power Plate (vibration training).


In the study, the experimenters had taken two groups and measured their respective 100m dash times before and after training. The control group did a 100m sprint, then no training and then retested their 100m sprints. The predictable result was no effective change. Meanwhile, the test group did a 100m sprint, then underwent vibration training by performing squats on the Power Plate and saw a statistically significant improvement in their sprint times. The issue, of course, is that the experimenters changed TWO variables and I would argue that it was in fact the squats that improved sprint times, not the vibrating plate. It dawned on me that as someone with no real scientific training and very little experience in the training industry could suss out an issue like this with a published paper, I wouldn't be able to trust anything else published in that journal since there was clearly no oversight or real vetting as to the quality or content of the publications.


Then, like Mr. Dubreucq I tried to find ways to relate the tales from the book into my own life in a practical way. To me, the takeaway is that each of us must maintain responsibility for the "outputs" we have control over. In the case of the scientists mentioned throughout Rigor Mortis, that means at the very least being aware of the wasteful and misleading information you're disseminating and limiting the negative impact of rushed, flawed or otherwise inaccurate papers. In my case, as a simple CrossFit trainer it means constantly testing and re-testing with a sharp eye to self-evaluation to ensure the best possible outcomes.

Comment URL copied!
Michele Mootz
November 28th, 2019 at 2:51 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I appreciate that Richard Harris is trying to bring this issue into the public eye. I unfortunately found myself reading much of this book without the shock and surprise that should have been my immediate response.

It SHOULD infuriate people to find out what is going on in our research and development industries (amongst others). However, I feel fortunate to have been involved in a community and culture that questions much of what is blindly received and believed as truth. I appreciate that we have been taught to have things PROVEN to us in order to get on board.

I agree with many of the comments below that parallels can be drawn from this book to many other industries (including our own). We are responsible for holding ourselves and our peers to a higher standard and serving as an example of what that standard looks like.

So...it begs the question- “how do we prevent ourselves from falling into the same trap?” Human nature and psychology play some role in our tendency to take the easy way out when it suits our needs. Holding ourselves to the standard we have worked so hard to achieve may be one of the greatest challenges we face.

Comment URL copied!
Austin Begiebing
November 28th, 2019 at 3:31 am

I felt the same Michele. The book shed light on the details of faults in peer reviewed science, but it wasn’t a shock. This community and Greg have long opened my eyes to those faults. I ended somewhat sad, questioning if it will ever be possible for the big companies funding these studies to place doing the right things for the right meaning over monetary motivation. Call me pessimistic, but I don’t think it is. It’s a broken system that will always be broken. We as trainers can simply do our best to educate people on the ground, in the box to open their eyes just as CrossFit has opened ours.

Comment URL copied!
Brunno Silva
November 28th, 2019 at 11:59 am

Felt the same Michelle specially because I saw everything for inside. I was happy to see in the book that there are a lot of people trying to change the culture.

Comment URL copied!
Eric O'Connor
November 28th, 2019 at 9:43 pm

Love your thoughts Michele and am also thankful that I'm involved in a community that is critical of almost everything. With that being said I was still shocked by the data in this book. I've known for a long time that research studies are flawed and can be largely inaccurate, but I also did not know the data on the reproducibility problems and the multitude of ways that can make studies inaccurate! Some of these issues I have never heard of or have thought of before....so my mind was blown. Definitely glad that Harris is bringing these issues to light and I was encouraged that there are people in the field that are trying to make a difference!

Comment URL copied!
Wes Piatt
November 29th, 2019 at 5:11 am

My same thoughts Michele! That’s all that kept popping into my head while reading.... we are just so lucky to have found this community

Comment URL copied!
Kristin Bowen
November 30th, 2019 at 1:26 pm

This is great Michelle! Thank you!

Comment URL copied!
Kolin Theede
November 28th, 2019 at 2:30 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

As I read Rigor Mortis, I recognized an ongoing battle in my mind. Both sides of the battle were fighting vigorously to become my final opinion of the text. On one side, an optimistic view that was fueled by scientist and researchers who recognized the importance of proper protocols, who want the best for the field. On the other side, a pessimistic view that found validity at every turn through countless examples of a broken system; a system that puts the individual before the common good. In the end, one side did not appear victorious. Instead, I settled on an old proverb that goes, “The road to hell is paved with good intentions.”


Much like the fitness industry before CrossFit, science is full of misinformation and bad habits that have been built over time, often unknowingly, by well intentioned people. As I read the book, the common theme of ‘the way things have always been done,’ kept resurfacing. Something as common in science as the use of lab mice is incredibly flawed, as talked about in numerous ways in the book. Yet, they are still the standard for research. From the outside looking in, it seems inconceivable that this could be the case. However, as I relate it back to the fitness industry, I begin to become more understanding. Before CrossFit, the standard was single joint, isolation movements and low intensity cardio. CrossFit put forth a superior solution for fitness, yet for many years there was little change in the industry. Much like the personal trainers wanted to protect their livelihood by casting aside CrossFit, many researchers look past proper protocols in order to find their next grant. While they might know the truth, they have a more compelling reason to keep things how they are. They are more interested in self preservation.

While it is obvious that the system is broken and the slow rate of change is discouraging, I still find great hope in the fact that their is awareness building. CrossFit spent many years on the fringes of the fitness industry, all while they held the key to it all. With a lot of hard work and patient, the truth has become mainstream and the fitness industry will never be the same. I believed that anyone who reads this book takes on the same responsibility as the early adopters of CrossFit. While we might be on the outside for now, our relentlessness will make a difference in the future. Science can change for the better. Conversations like this are an incredible stepping stone on this journey and I am cautiously optimistic about the future of science.

Comment URL copied!
Michael Marrone
November 28th, 2019 at 2:27 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

My takeaways from Richard Harris’ “Rigor Mortis”


The idea of the “power of suggestion,” first introduced in Harris’ discussion of transdifferentiation (p. 24), and which is subsequently echoed throughout the book, is not unique to the medical community, but it undoubtedly carries greater risk (financial waste, loss of life, etc) in the search for successful biomedical research than in other endeavors. Consider an emergency or combat incident. The first call is generally errant and masked by erroneous information. It takes an internal discipline to pause and think, in spite of the overwhelming excitement and adrenaline, whether that be from a potential groundbreaking biomedical breakthrough or a catastrophic emergency. In this sense, to fix the fundamental problems plaguing the biomedical community, we must not only amend a flawed process, but also engage with the personalities behind the process so as to develop a measured and reasoned response to biomedical research. In every sense, this discipline is the rigor to which Harris speaks. I think amongst CrossFit trainers, there requires a similar commitment to discipline, one that mitigates the misconceptions of traditional approaches to fitness, and simultaneously promotes and protects the CrossFit brand. CrossFit Trainers must recognize that coaching is not merely a matter of showing up and teaching (although that is part of it), but a much larger commitment to a profession that, when executed with intent and purpose, instills within the community a discipline and motivation to achieve movement and standards that carry over to every facet of our well-being and livelihood. This discipline should guide Trainers in their precision to which they execute the six criteria of effective training (teaching, seeing, correcting, group management, presence and attitude, and demonstration), and all that follows: the ability to adeptly convey progressions, meet timelines, inspire health and fitness amongst the community, entertain those they lead, among many others.


Of the several themes that run throughout the book, there is one which deserves attention due to its similarity to exercise science related research and CrossFit’s groundbreaking efforts to change the nature of this hampered system. Drawing from Begley’s “six most common preventable failures,” (p. 27) one of the key elements of irreproducible biomedical research is that “sometimes researchers cherry-pick their best-looking results and ignore other attempts that failed.” This seemingly parallels some of the issues currently facing exercise science related research. In paraphrasing a lecture by Dr. Nathan Jenkins during the 2019 CrossFit Trainer Summit, current exercise science related research tends to limit itself as to the gender, age, and fitness levels of its test groups, as well as generally limits the intensity levels of the tests, typically seeking to evaluate (and define) “fitness” largely as “cardiorespiratory fitness,” and namely through the evaluation of one’s VO2 max (maximal oxygen uptake). The tests ultimately forgo the research of high intensity regimes, as well as largely evaluate, or “cherry pick,” a cohort of individuals that fit traditional norms: young to middle-aged males with average body types. In essence, the broader exercise science community has failed to address the problems of chronic disease by failing to adequately and comprehensively provide measurable data across a broad spectrum of factors, namely data to illustre the effects of constantly varied functional movements at high intensity on all genders and ages. Fortunately, CrossFit, through the likes of researchers such as Dr. Jenkins, and through the community writ large, is making impressive and groundbreaking waves in the exercise science industry. Ultimately, by developing evaluations based on CrossFit’s guiding principles and definition of fitness (work capacity across broad time and modal domains), CrossFit is leading exercise science related research in forging “rigorous and scientifically sound” methods to become the leader amongst exercise science practitioners. Should the biomedical field face similar groundbreaking efforts to challenge the stats quo, the combination of CrossFit’s methodology and this renewed biomedical research will be a robust force in conquering chronic and rare diseases. (For more information from Dr. Jenkins, visit: https://journal.crossfit.com/article/science-jenkins)


As discussed by Harris, there are undoubtedly two factors, which may inhibit the speed to which biomedical research can keep pace with its growing demand: time and money. With this in mind, Harris illustrates the importance of private funding to support and promote rigorous research. In the face of the National Institutes of Health (NIH) rejecting over 80% of grant proposals (p. 62), the writing process for which takes endless hours, waiting on federal grant money is not an answer. Instead, individuals, private organizations, and communities must recognize the problem at hand, and rally to fund research. Grassroots efforts such as CrossFit Health are just one of the ways to educate individuals on the issues of funding and lack of rigor facing the biomedical community in hopes of promoting its rehabilitation.


Of course, recognizing rigorous and worthwhile research efforts is the challenge. From an over-reliance on mice (p. 50) to a general uneasiness amongst the scientific community to challenge and criticize the research of those who are responsible for granting future funding (p. 37), there are a range of concerns that obfuscate rigorous research. Chief amongst these issues is identifying talent. The current system pressures, if not forces, researchers to publish work - “though not necessarily the most careful or the most important” - in elite journals (e.g. Nature), in an order to gain funding and support by universities (p. 174). Yet, this approach only perpetuates an industry that values “flashiness” to thoughtful, rigorous, and inspiring research. The issue at hand is twofold.


First, there is an over-reliance on journals to determine talent. As described by the former executive editor of Nature, “The scientific community outsources to [journals] the power that they haven’t asked for and shouldn’t really have” (p. 177). These journals essentially determine “scientists’ fates when choosing which studies to publish.” This in turn leads to the second issue. In order to be found, researchers know they must produce and publish in journals. This leads to publications that lack rigorous research efforts in order to hastily publish research so as to found amongst the talent. As described by Randy Schekman, a Nobel-prize winning scientist, “It’s hand in hand with the issue of reproducibility because people know what it takes to get their paper into one of these journals, and they will bend the truth to make it fit because their career is on the line” (p. 177).


Although Harris does not provide concrete methods to amend the aforementioned issues, he does present soft methods to mitigate the risks posed by hasty and unsatisfactory biomedical research. In essence, this means addressing the deep structural and funding problems found in biomedicine. First, there needs to be a change in culture amongst the biomedical community. The community cannot fear - or admonish - failure, but rather, must embrace honest research so as to not repeat the same failed (but at least repeatable) approach. Second, and as discussed earlier, the community must engage on and foster discipline and integrity in its research, a factor much easier said than done considering the current system’s fight for funding. With that in mind, the last factor that may address the issues plaguing biomedical research is to revamp the “imbalance between the money available for biomedical research and the demand for it among scientists” (p. 188). As discussed before, one method to promote increased funding is by spurring private investment. Other approaches include encouraging universities “to establish an endowment to fund key professors’ base salaries to reduce the do-or-die scramble for research dollars,” and encouraging states to enhance funding efforts.


Whatever the method and means for restructuring the system of biomedical research, CrossFit Trainers, and the community at large, must understand they too have a role to play in advancing research and providing Big Data and evidence as to how constantly varied functional movements at high intensity hedge sickness and mitigate chronic disease. The fight to save lives is not medicine or fitness, but a mutual approach, which when structured and balanced, can adeptly reduce healthcare costs and create stronger and fundamentally safer communities.

Comment URL copied!
Kristin Bowen
November 30th, 2019 at 1:25 pm

I enjoyed reading your thoughts, Michael. I was also immediately struck at the fact that researchers will cherry-pick their best results and ignore ones that failed! Yet another example to show how little I knew about the methods in these experiments. I love the comparison and your explanation on the parallels to exercise science research. This helps connect the dots; thank you for sharing!

Comment URL copied!
Nathan Jenkins
November 28th, 2019 at 2:09 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

All the CF Seminar Staff commenting on this thread suggests to me that this book was an assigned reading? I haven’t read it but have heard of it. This post and the comments have catapulted it to the top of the reading list. Thank you all for the perspective here.

Comment URL copied!
Cherie Chan
November 28th, 2019 at 2:19 am

Nathan - it’s pretty cool to all be reading the same book each month. Check out our CrossFit training IG soon for next months and join us, it’s a CrossFit HQ book club.

Comment URL copied!
Nathan Jenkins
November 28th, 2019 at 2:24 am

Cherie, thank you! That sounds amazing and I would love to join you all in that. This particular book hits close to home for me... just ordered.

Comment URL copied!
Michael Marrone
November 28th, 2019 at 2:28 am

I paraphrased you from the Summit, so if you want to take a look and add any of your thoughts!

Comment URL copied!
Lachlan Learg
November 28th, 2019 at 9:21 am

It's great to get more people involved in the discussion. I'm sure you could offer an insightful perspective and some knowledge. Look forward to hearing your opinion of the book.

Comment URL copied!
Peter Shaw
November 28th, 2019 at 1:56 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

My first reaction when reading this was “Wow, what have we gotten ourselves into? More people need to know about this.” But, it’s clear that at least a few people are thinking about these problems. So why are the issues so pervasive? The root cause seems to be something that influences corruption beyond just the scientific and medical realm — money.


As Richard Harris mentions, everybody goes into science and research with good intentions. The goal in research is always to discover solutions to the problems of our world (be it theoretical or practical). But at one point in their careers, many scientists are forced to make decisions that dictate the success, survival, or demise of their laboratories and life’s work.


Its human nature to make poor ethical decisions when backed into a corner and scientists fighting for research money are no different. However, it is during those moments that we must remind ourselves that Truth always prevails… Maybe it’s the fact that strength and courage are needed at times to speak the truth. Maybe it’s the rarity of these attributes partly contributing to “The Mess”.


I have no doubt that there are those in the field of biomedical science who believe they are contributing to a wholesome cause, but are in fact “creating worthless cures” and “wasting billions”. However, as a scientist it is your responsibility to be the master of your domain and know all the strengths and weaknesses that come with it. Being a true scientist means searching for the Truth, and not believing yourself if you think you’ve found it.


As CrossFit Trainers we can learn from the problems in the biomedical world by standing by our clients and educating them on what we know works — CrossFit combined with proper nutrition. We should also always question our methods and test new conjectures, but then use measurable, observable, and repeatable data to help build our conclusions. At times it may take courage, but the Truth never dies.

Comment URL copied!
Joe DeGain
November 28th, 2019 at 3:20 am

Courage... Well said, Pete.

Comment URL copied!
Cherie Chan
November 28th, 2019 at 1:36 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It was challenging to articulate a comment on Rigor Mortis. It was a difficult read for me. I love reading but feeling angry, disappointed and helpless are not enjoyable emotions when reading. With that said it made me think a lot about human behavior and what creates change or habits.


While it is difficult to admit, we are all driven by moving toward reward and away from pain. (Likely why this was hard to read and maybe why more don’t). We have created or been a part of a system that rewards “bad” science and causes pain for “good” science. Not in all cases of course, but in many referenced in this book. Until we change what we reward we will continue to be frustrated and evade the truth.


On a different slant, this also got me thinking a lot about how physiology is not as easy as isolating variables (that you hope are untainted). That It’s a system that works as a system not as independent variables. It takes the whole systems to get sick and the whole system to heal. On top of that, it takes a lot of time. Sometimes decades to have a result. A concept that seems insurmountable and I applause those working hard to provide solutions with good intent.


It’s a bigger problem then I feel capable of attacking. It’s why I believe we need to strength critical thinking skills and common sense. I’m grateful there is a platform here to provide this information.

Comment URL copied!
Dafydd Dennis
November 28th, 2019 at 4:24 pm

Cherie I echo your statements. Firstly, this book was indeed a difficult read. But rather than just simply due to the scientific vernacular, referenced studies or subject headings...the overwhelming number of examples of bad science.


Should the reader delve into each and every reference (and every time that they are utilised) in an attempt to quantify the reasons for the apparent failure. Or, look at the person or should I say character of those conducting and look for confirmation..."human beings have vested interests".

Comment URL copied!
Kristin Bowen
November 30th, 2019 at 1:17 pm

Thank you for this honest comment, Cherie. I admit I felt the same when reading. It really opened my eyes to how much my head has been in the sand and the blind trust I have always given to the scientific community. This quickly led to frustration and helplessness. However, reading your comment as well as all of those written as been a wonderful way to finish this book. While this problem is far too large to have an easy solution, it's the smaller changes we can do as individuals on a daily basis as well as slowly educating those around us that can make the biggest impact.

Comment URL copied!
Matthieu Dubreucq
November 28th, 2019 at 1:36 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

It is impressive the amount of similarities between the world of science and the world of training. I talk about the facts illustrated in the book like putting short term profit first, racing to have the biggest headline and so call “next big thing”. It seems that thinking about your ego only is the lazy science way some have chosen. I am shocked that scientists don’t read studies, it is just about publishing it. That it is more important to show you wrote a lot of article then to have one article that actually solves something. That doctoring data is such a big thing.


So what can a trainer learn from Rigor Mortis? I believe that the main message is to become better. To fight the lack or rigor in science (and the training world) by fighting the cause : ignorance about the process of publishing a study. How do we fight ignorance? By education. How do we change the future? By learning from the past instead of hiding it. There is a huge need for education. Just the section about the lack of knowledge of basic statistic is dramatic.


I realized I need to educate my self to make better decision about what I believe, what I read, what I trust and so do you!

Just like studies, it is pretty fair to say that most of training out there rely on broken science and is just a push to serve some big companies that want your $ and have no intrests in your health. How do I know? I am an expert in the field. So the first lesson that you can learn from this book is to become an expert in your field, in the research practice of your field.


Who is funding the study your are reading? How is the funding process working? What are the usual bias? What are the main shortcut scientists take? Finally what source can you trust?


The message of hope is that the scientists who will succeed (solve problems) in the future are the one that don’t fall in the trap of short term fame. Don’t let big pharma pay their bill and tell them what to “find”. Well it is the same in the field of training. Look at what CrossFit is doing! Deliver the truth, don’t tie yourself to a big Soda giant and never hide your mistakes.


Like coach Glassman has always said : market always rewards excellence. Don’t let yourself get blinded by the short term success go for excellence by becoming an expert and write the future. Don’t let sloppy science and trainers negate the main population that need a long term fix for their health problem. A new discipline is born!

Comment URL copied!
Cherie Chan
November 28th, 2019 at 1:44 am

Thank you Matthieu I needed to hear the half full perspective!

Comment URL copied!
Joe DeGain
November 28th, 2019 at 3:25 am

Nice questions for anyone wanting to evaluate if they are truly an expert in the field of research...

Comment URL copied!
Brentnie Kincaid
November 30th, 2019 at 12:07 am

I had the same thoughts, Matthieu! The world of health, fitness, and nutrition are filled with the same lack of scientific rigor. We need to be better, pursue excellence as coaches and in educating our communities. We can be more proactive in preventing chronic disease. I am so thankful that I have found this community of relentless, forward thinking individuals who are changing the world. The more we change lives in our affiliates and communities, the less people will have to depend on biomedical research.

Comment URL copied!
Gregory Kerschbaum
November 28th, 2019 at 1:34 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

After reading the book Rigor Mortis, by Richard Harris, a quote stuck out to me. “Scientists need both ambition and delight to succeed” He goes on to state “Without curiosity, without the delight in figuring things out, you are doomed to make up stories.”


Throughout the book, I noticed a number of examples where a scientist or a team would refute previously held beliefs, only to have their view quashed by journals or members of the field. In one such example, Mark Davis was trying to publish a paper on immunology that was contra to current theory. His attempts attracted strong push back, but it was not for any scientific reasons, the reason stated was that he was hurting the field, and going to setback research 10-20 years. To me this kind of thinking is completely delusional, because you can’t set research back if it wasn’t based on truth in the first place, and if you are going to dismiss a paper, it needs to be on scientific grounds. If I bring this same idea into the field of fitness, if someone came out and said they had a better way to produce increased work capacity across broad time and modal domains, I would question it, and make them prove it, but I would not immediately try to stifle the message. If there is something better out there I want to know about it, and I want to steal it to use. It goes back to ambition versus delight. A scientist needs to delight in the field moving forward, They should be excited when someone questions their research, they should look critically and make sure it is sound, but then be excited that those criticisms are what move the field forward. They provide a vector check, a way to course correct when you have gone astray. This was something that resonated very strongly with me, as it just seems like our society is moving away from truth. Truth needs to be our guiding light, and I really feel that it is vitally important that we prioritize scientists that are delighted about truth and delighted about moving scientific research forward, the same way I am delighted and excited about fitness. I don’t have all the answers to turn this around, but I am really glad that we have organizations like CrossFit, who are not afraid to speak up and to give a forum for truth. I think its the first step to getting the information out there and providing a platform that is not blanketed by false views and dollar signs

Comment URL copied!
Natosha Haydock
November 28th, 2019 at 1:32 pm

Gregory I really like this quote as well! It is a simple, positive statement that you hope many truly live by. I see it with Crossfit staff and coaches

Comment URL copied!
Jobst Olschewski
November 29th, 2019 at 4:22 pm

Gregory, I like the quote you picked, as well as your fitness analogy. Like for you, it seems very intuitive to me, to first question a statement/hypothesis before using it for myself and/or using it as a basis to move forward.

I think this might also be applied to what we do as trainers: Before we teach something, we want find out if or how it works: As a teacher, in a class, "without the delight in figuring things out, [we] are doomed to make up stories."

I agree in part with Joe [Masley] who mentioned the structural obstacles of missing "uniform standards", but what you are pointing out are in particular the cultural issues in the biomedical community that are exposed: an apparent lack of pride for "doing it right". It is one thing not knowing how to do it right, but what makes it worse is that in many cases there is no real interest in doing so: Rather, grants or reputation through journals is being ranked as more important.

As Richard Harris writes in his first chapter "There is little funding and no glory involved in checking someone else’s work.", describing one of the dilemmas in the biomedical research community - when the attitude should be: "It goes without saying that we check anyone else's work first".

I feel really blessed to be part of the CrossFit community, where I see a different culture: The delight of figuring out how to Air Squat and the pride of doing it right. And everything else can then be built on this solid foundation. And if someone does not get it right (yet), at least they are trying their hardest to get there. An amazing culture that is "doing all the right things, for all the right people, for all the right reasons."

Comment URL copied!
Brunno Silva
November 28th, 2019 at 1:12 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

During a lot of moments in the book I thought some paragraphs were written by myself

I can explain:

I’m a former academic who went for a Master’s degree in Physical Activity Sciences in 2012, in Brazil, just after my under graduation in physical education. Same manner author mentions in pg 163, I choose academic world because I was excited with the idea of make a big impact in exercise science field. Same manner Harris describes I ended up frustrated.

Personally my frustration was because I realized Nobody was really worried in create a social impact,that means, apply the knowledge discovered to improve either the act of professionals when prescribing exercise or the quality of people’s lives through exercise.The motivation revolved arround publish as many papers as possible in a Journal with a high impact factor no matter if the research was relevant or not. I remembered I was guided to a research that it wasn’t my original Idea (and I don’t think it was relevant at all) because it had a higher “potential of publication”.

The requirement of publish a certain number of paper in “high quality” journals made the advisors ask to students to rush when collecting data. But there were a problem. The responsible for research either didn’t know or had lost practice in how to collect data. They trusted the data being collected for inexperienced researchers who were being rushed. So they could finish the experiments faster. Needles to say probably a lot of studies are not reproducible.

It’s a cultural problem that is far from finish, like reported by Harris. During all process I was told that I had to eat crow and not go against my advisor because they could cause problem to me in the future. Students have to deal with the pressure of the advisors and the large amount of responsibilities suddenly they have to do (some of them which should be advisor’ responsibilities). Under pressure it’s not rare to see a lot of students get sick. I got Graves Bocio during my Masters, a immune desease related to emotional imbalance and there are a lot of cases of psychological problems in students of Masters and PhD (there’s an article by folha de São Paulo that reports that). Guess what? This same students have to do important researches and there’s no way this can be a good job.

To finish my comments, I got very happy seeing that I’m not alone in my thoughts about cultural problems in academical field and I add one more problem to the mess. The problem of bad leadership created by the advisors, who are not willing to teach less experienced students a step by step how to make good science.

Comment URL copied!
Jessica Pilling
November 28th, 2019 at 1:06 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions



It was refreshing to know that most individuals that go into biomedical research honestly want to change the world, they are curious and want to figure things out ("...it's the unexpected ideas that often propel science forward") and defeating to learn that money skews their passion- once they get into it, it's now a choice between doing what is right and doing what will keep the money coming in and their careers advancing- even when they know the research is sloppy and not correct, they publish any ways out of fear of losing funding or the death of their career.

As someone who has lost a loved one due to cancer, the areas of cancer research in "Rigor Mortis" interested me the most and left me the most disappointed. Looking into drug development 9 out of 10 fail because the science isn't rigorous and therefore unreliable, That's like saying "9 out of 10 airplanes we designed feel out of the sky" and there's so little advancement in biomedicine due to this that it's predicted that by 2040 there will be no new drug advancement. What fired me up is that there's millions of dollars spent on cancer research, specifically breast cancer, with little to no advancement. And unfortunately this makes sense when the cells they thought were breast cancer had been cross-contaminated. And once this was discovered, the research continues to be repeated and done so incorrectly! Specifically looking at melanoma cell line when the research was actually studying breast cancer and then having over 1000 papers since published with the contaminated cells looking at the wrong cancer.

Learning the dire facts has further instilled in me as a trainer to focus on the preventative measures for myself, my family and the athletes in my affiliate. Move more than sit and "Eat meat, vegetables, nuts and seeds, some fruit, little starch, and no sugar."

Comment URL copied!
Joe DeGain
November 28th, 2019 at 2:35 am

Nice, Jessica.

The evidence of the crookedness is there.

The practical advice in your last paragraph brilliantly expresses an elegant solution.

Comment URL copied!
Joe Masley
November 28th, 2019 at 1:05 am
Commented on: Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions

I come from a military background and currently serve as full-time firefighter outside of coaching CrossFit. These professions all rely on specific methods of training focusing on rigorous uniform adherence to standards and procedures. These uniform standards and procedures make it easy for anyone inside or even outside these professions to understand performance expectations and how tasks are to be carried out. Deviation from these procedures are relatively easy to spot and the individual cultures around these professions pride themselves on virtuosity in the basics and upholding the standards.


That said, my perception of the biomedical research field and its scientists prior to reading Rigor Mortis was that they, too, held themselves to strict procedures and also prided themselves on upholding rigorous standards; especially since many of these studies help to develop drugs and medical practices charged with protecting human lives. I was naive in those perceptions.


The three things that disturbed me most reading this book were:


(1) There is no uniform method for conducting studies that is inculcated into young scientists during their graduate and doctorate programs nor is there a uniform standard that all scientists follow when they start their careers that is simple to follow. It seems as if the scientific method is an after-thought and rather it is up to individual labs and institutions to come up with their own methods as it suits them.


(2) A dangerous culture of fear of being incorrect exists that perpetuates a cycle of cutting procedural corners and molding data to fit predetermined conclusions in order to achieve correct-looking and flashy results. There seems to be little pride in upholding rigorous research practices.


(3) Private, for-profit journals represent the gold-standard in disseminating information and seem more concerned with flashy titles that will sell publications rather than iron-clad, correct data.


Despite the above, it is encouraging to see that not all scientists fall into these categories and that many needed changes are happening within the profession by those dedicated to excellence.


I think the best answer to these problems can be summed up in the paragraph at the end of page 214 continuing into page 215:


"The deepest challenge in realizing the potential of precision medicine is changing the underlying incentives in biomedical research. That means re-engineering the culture. The question is how to do that. Step one is to make sure the problems and the perverse incentives are well understood. Step two is to figure out how to create new incentives for scientists, universities, and funding agencies." 

Comment URL copied!
Brunno Silva
November 28th, 2019 at 1:18 am

Amazing Joe. I would add to that a culture of good leadership (I talk more about that in my comments). Unfortunately it’s not a place you are free to make mistake, be corrected and move forward learning with that mistake.

Comment URL copied!
Michael Marrone
November 28th, 2019 at 2:32 am

Joe. I agree. What I find perplexing is how do many industries - including the aforementioned biomedical sector - are so perplexed and offended by failure. Also coming form a military background, and now working in the private sector, I’ve come to realize how unique the military (and likely law enforcement and emergency services) are in accepting failure in training and preparation. This must have something to do with the tremendous stakes that are om the line in the execution of their mission in real time, so accepting failure (and more importantly, learning from failure) is so essential to effective operation when it counts. If only biomedicine and many others realized this too.

Comment URL copied!
John Brown
November 28th, 2019 at 4:40 am

Joe,


I have a few quick questions:

In the Corps, were your SOPs the same as all of the other divisions or were they specific to your unit?

On the fire service, are your procedures the same as a fire service unit in say, Colorado?


My guess is that the answer is "no" to both. My concern is that different doesn't mean wrong and also that if we do not do something different then we will always fall short of the innovations that carry us into the future.

Comment URL copied!
Joe Masley
November 28th, 2019 at 12:16 pm

John,


That is correct - there were specific SOPs unique to each unit in the Marines and tactics for a city fire department in Boston are different than a suburban town in Colorado. I agree that strict adherence to doctrines and dogmas inhibits innovation and progress and different doesn't mean wrong.


My observation for the book is that young scientists have no methods or procedures to start with at all, it seems methods, procedures, and standards when it comes to the finer details are afterthoughts. For example, all units in the Marines followed certain procedures like "weapons, gear, self" when it came to cleaning after an operation or exercise. In the fire service, regardless of town, you will see each company meticulously combing over their apparatus and equipment in the morning to ensure proper function and safety. There are at least basics that the majority follow when it comes to procedures that dictate the safety of their teammates or those they serve - those units that don't are looked down upon by their peers and usually called out to correct their behavior.


I think it's important to evolve and find better ways to execute on procedure regardless of profession, but there needs to be a solid, common and understood starting point and I hope that the biomedical field is able to find that first before moving into flashier techniques.

Comment URL copied!
Richard Gonzalez
November 28th, 2019 at 2:55 pm

John/Joe,


I think the SOP analogy fits, but it gets muddled in academic and private funded labs.


In the Marine Corps, there is warfighting doctrine, Marine Corp Orders, Battalion orders and TTPs.


In academia there is the peer review process. From my understanding the peer reviewers don’t necessary check the validity of the paper, but where it follows some guidelines; Does the paper fit the journal, formatted correctly (Intro/Lit Review, Methods, Results, Analysis, Data, etc).


However, I don’t believe young scientists are unaware of scientific rigor. In fact, they are likely idealistic about it, until the reality of publish or perish hits.


In both of these cases, human factors will always get in the way of what “should” happen. In my Marine Corps experience, in less than high speed units, I’ve seen many corners cut, sometimes due to expedience, sometimes due to mission priority, and often time because of lazy or lack of leadership. And Joe, like you mentioned above, this book is just one example of the community calling itself out (as John said in an earlier reply, policing it’s own).

Comment URL copied!
John Brown
November 28th, 2019 at 5:51 pm

Rick, Joe:

Agreed on all with one caveat. I have a very close friend who is wrapping up his PhD in neuro pharmacology(a real dummy for sure) and we talk all of the time about the process. He is beholden to the person running the lab that he works for... so it isn’t that new or young scientists are without procedures, it is that they learn from the people that accept them into the program. He has told me on several occasions that he will not run his lab the same way this person does, so maybe there is hope.

Comment URL copied!
Nolan Mooney
December 12th, 2019 at 6:34 am

Joe, I too have a military and firefighting background and I agree with the need to establish consistent procedures but much like John already mentioned, I don’t believe it is a case of a lack of existence of said procedures. SOP’s were different between platoons in my unit/fire house and although I do not have a background in scientific research, I do not believe that there are not procedures present, nor do I think the book is trying to say that. I believe the difference is that in our old jobs, if we failed to adhere to specific SOP’s there was potential for immediate and sometimes life threatening repercussions. A failure to properly execute a JMPI could result in a catastrophic failure of your chute = immediate and final consequence for the paratrooper. The consequences of not following a particular procedure in an experiment may not have the same immediate impact(not that a lack of results/reproducibility isn’t important) which may create enough separation between the researcher and potential patient, and remove some of the pressure to apply the required “rigor” or in paratrooper terms “attention to detail.” I just don’t believe that the majority of people in the lab are purposely foregoing procedures. I believe the purpose of the book was to shed light on a very real phenomenon that is happening and that we should be cautious to jump to conclusions about everything we read/interpret, INCLUDING this book! I too am guilty of “believing “ an argument/ document simply because it was peer reviewed in a scientific journal. Rigor Mortis contained many citations( which is fantastic) but I know that I did not follow each one and review each statement nor do I have the scientific background to determine wether all the statements and claims were legitimate/true. I was not upset or angry when reading this book but intrigued. It made me more curious and motivated me to continue to read and evaluate everything critically. After critical analysis, including asking for help when my lack of knowledge prevents my understanding the subject, I will make my own educated decision. I think we as Seminar Staff need to be wary of jumping head first in ANY direction and just as we discuss in our seminars, to question ALL information we come across.

Comment URL copied!