I remember some of these tools and ideas for analyzing scientific literature being taught to me in school, but they are quickly lost if not used regularly. In undergrad we receive an overview of the basics and are sent on our way into careers in science. From there, if you are working with a group of intelligent professionals you will regularly take part in Journal Club meetings to discuss literature. These can be a great way to encourage critical thinking about science.

Unfortunately, not everyone has time for Journal Club meetings and we are all influenced and touched by science in some way, shape, or form. So it is useful to have a few tools we can use for quick analysis over coffee.

Thank you!

Thanks for the tools to help us read a study and know if it is flawed or legitimate.  

"In Defense of Food" by Michael Pollan provides some interesting historical context on how these types of studies came to be the norm, especially in regards to the analysis of nutrients instead of the food itself.

It’s deeply discouraging how often, as Feinman shows in a single case here, the most faithful interpretation of the data is not the one that ends up in the Abstract. It’s simply not feasible for anybody - even an academic - to read every paper in even a narrow field with the diligence required to render independent judgment. What share of the literature has been broadly interpreted in ways the data did not support?

One consistent theme I’ve noticed working with and talking to older academics like Feinman - the sort who did much of their work before computers made statistical analysis and reanalysis easy - is the higher bar many of them set for data analysis and discussion in papers. When data were harder to gather and analyze - more precious - there was a greater emphasis on, and expectation for, extracting all possible interpretations out of them, including those that may have unintentionally arisen via the study methodology. In this case, as Feinman notes, simply changing the independent variable (from calories to carbs across all phases) reveals a relationship with triglyceride levels more consistent than the relationship the authors focused on.

I suspect much of the “noise” in nutrition research could be cleared if this sort of comprehensive interpretation were universally applied. Very often, if a study arrives at a surprising conclusion the reason can be found in the methods or the interpretation of the data. (This points to one of the perils of meta-analyses, which Feinman critiques elsewhere…these nuances are lost when the pure quantitative results of papers are analyzed together.)

Yet coming to this conclusion requires spending a couple hours with this paper - an approach that is useful for individual studies, but not scalable. What’s the solution? I don’t know, beyond finding a few sources whose interpretation you trust and who you believe interpret the data both critically and charitably as warranted. In the meantime, when we can, it’s good practice to look for ALL the inferences that may be consistent with the data, and recognizing that the authors may not have made the correct inference themselves.

Feinman and Volek’s analysis is worth reading directly, twice if possible.
https://nutritionandmetabolism.biomedcentral.com/track/pdf/10.1186/1743-7075-3-24

Richard Feinman writes an incisive critique of modern scientific literature before offering a list of helpful guides for discerning whether a scientific study is flawed or legitimate. To preface his critique, he includes an epigraph from Shakespeare’s Macbeth, which got me thinking. I appreciate any excuse to return to Shakespeare, so I started reviewing the context of the quotation Feinman cites and the doctor’s role in the play. In doing so, I’ve discovered that Lady Macbeth, the patient to which the doctor refers, may be read as an allegory for the crisis of modern scientific literature.

At the heart of Feinman’s critique is the sense that modern science has become an echo chamber wherein popular theories are repeated ad nauseam until they become dogma. The barrier to entry into the echo chamber is a willingness to repeat the popular theory. You must “conform to the party line,” Feinman says. If you don’t repeat and confirm what others have argued–right or not–you don’t even get to join the party. Enter Lady Macbeth.

Lady Macbeth might be read as an allegory for this problematic situation in the sciences. When Lady Macbeth begins her sleepwalking with repeated ravings, Macbeth grows worried and consults a doctor. The doctor, learning of Lady Macbeth’s symptoms and determining them to be a sickness of the mind rather than the body, speaks the lines Feinman cites: “The patient must minister to himself,” to which Macbeth replies, “Throw physic to the dogs, I’ll none of it” (5.3.48-51). In other words, Macbeth is exasperated by the doctor’s inability to help him. 

If you flip back a few scenes in the play to 5.1, it becomes apparent just how insidious the doctor’s inability to help is, and what is at the root of the failure. There, the doctor observes Lady Macbeth’s symptoms for the first time. “Unnatural deeds do breed unnatural troubles,” he says (5.1.70-1). “Infected minds to their deaf pillows will discharge their secrets ... My mind she has mated” (5.1.71-8). The Penguin editors gloss “mated,” as a clipped version of “checkmated” that means to imply “defeated,” i.e., Lady Macbeth’s peculiar symptoms have defeated the good doctor’s training and left him baffled. This valence is certainly present in the line, but so is the more modern meaning of “mated,” which refers back to the doctor’s reference to breeding: “Unnatural deeds do breed unnatural troubles.” 

In continuing the image pattern in this way, the doctor suggests that the “unnatural troubles” that plague Lady Macbeth are reborn in his own now-corrupted mind. His mind, like hers, is unsettled. Further evidence of her infection or impregnation of him (near-interchangeable terms in the doctor’s speech) is apparent in his parroting of her language patterns. Lady Macbeth is given to rambling repetition–“To bed, to bed ... Come, come, come, come” (5.1.65), as well as her infamous, “Out, damned spot! Out, I say,” (5.1.34). The doctor’s speech remains more eloquent than Lady Macbeth’s ramblings, but his repetition of “unnatural” and later, “God, God forgive us all” (5.1.74) is suggestive. The cycle of repetition is a symptom of the sickness that upsets all order and reason for Lady Macbeth, perhaps for her doctor, and certainly for modern scientific literature. 

The cure? In Macbeth’s words, we have to “raze out the written troubles of the brain” (5.1.44). Feinman offers some excellent advice for beginning that process when engaging with scientific texts.