Question
How significant are the observable flaws in the design, analysis, and reporting of nutritional epidemiology?
Takeaway
Looking solely at whether the analysis and reporting of nutritional epidemiology are supported by “good scientific principles,” John Ioannidis argues the field is fundamentally flawed. This form of research regularly and predictably produces implausible correlations, which are then described using causal language that influences public thought and policy. Given the complexity of the modern diet and the many correlations between different dietary components, it is difficult if not impossible to isolate the health benefit (or harm) of any individual dietary factor through observational analysis.
In this 2018 commentary, John Ioannidis argues the reporting and analysis of nutritional epidemiology are fundamentally flawed. He writes, “The emerging picture of nutritional epidemiology is difficult to reconcile with good scientific principles. The field needs radical reform.”
Ioannidis reviews some of the dramatic — even implausible — claims that have emerged from epidemiological research including:
- Eating 12 hazelnuts per day will extend lifespan by 12 years; drinking three cups of coffee per day will have the same effect.
- Eating one mandarin orange each day will extend lifespan by five years.
- Consuming one egg daily will shorten lifespan by six years, and two daily strips of bacon will shorten lifespan by a decade. The latter effect is greater than that of smoking.
The authors of such studies generally describe these correlations using causal language, such as “optimal consumption of risk-decreasing foods results in a 56% reduction in all-cause mortality.” Media reports often use causal language even when the authors do not.
Ioannidis argues these effects do not represent the true impact of individual food items but rather the cumulative biases present in this sort of research. Consumption of each food item is correlated, positively or negatively, with hundreds of others, and with an unknown number of nondietary factors that affect health (e.g., geography, education, income, smoking habits). We do not have a sufficient enough understanding of how these factors correlate with one another to correct for their impact, and given that there are over 250,000 foods in the U.S. food supply, fully understanding these correlations may be impossible. As a result, a single correlation drawn from nutritional epidemiology reflects any true effect that food item may have and the various confounders it is correlated with to an unknown extent.
Given this ambiguity and the thousands of possible correlations drawn from any epidemiological data set, it is easy for researchers to favor the correlations most consistent with their existing beliefs about diet and disease. Meta-analyses become “weighted averages of expert opinions” rather than objective assessments of the data. This is exacerbated by the fact that analyses of a single data set can be split across hundreds or thousands of papers; the Nurses’ Health Study alone has yielded more than 1,000 publications. When each association is reported in isolation, it is given exaggerated significance and not seen for what it is: one of thousands of possible associations, each confounded, which could have been drawn from the same data
At minimum, improving the quality of nutritional epidemiology will require publicizing the data from large trials and discarding any past claims in public health, media, or policy drawn from epidemiological studies’ causal claims. Future reform will require greater data transparency, data sharing, and a shift in how this research is presented and analyzed within and beyond the research community.
Comments on The Challenge of Reforming Nutritional Epidemiologic Research
Another superb Ioannidis article. Nutrition epidemiology is profoundly flawed, and as I recently described here, the only safe way of approaching a paper that reports that breakfast is healthful is by asking yourself "how are these lying b*stards lying to me?" By asking yourself such a question, pro-breakfast papers are actually fun to read, like doing a crossword or a sudoku game. Breakfast cannot be healthful for normally-fed individuals in the west, so the game is to work out how the paper has been constructed to claim it is. To avoid the downside, it's best not to ask "why are these lying b*stards lying to me?" because there's only so much disillusion with our fellow humans that we can take without becoming fully misanthropic.
Where breakfast goes, so does the rest of nutritional science, which is broken almost beyond repair.
How to fix it? Well, we could demand that all nutrition papers should be on-line, open access and open to comments like this. And we could then commission at the authors' or the journals' expense responses from capable epidemiologists like Ioannidis, as well as from biochemists and other relevant scientists (the only disqualification being if they'd ever published a nutritional epidemiological paper themselves) to critique the papers meticulously.
That might make a start.
There has always been epidemiology; it just used to be called superstition.
Numerology, oracles, mystics, astrologers have forever relied on epidemiology: making connections between two variables that may or may not be connected in order to claim a specific conclusion.
Don't sign contracts when Mercury is retrograde. Werewolves come out when there's a full moon. If the number 7 appears three times in a day you will get good news.
The Romans had a special practice call augury, which was the interpretation of bird sightings as a means of interpreting news. The augur, person charged with making these connection, would read the auspices -- which comes from the latin auspicium, avis=bird + specere=look and share the conclusions with the people. We still use this word, auspice, to mean good fortune or protection.
Epidemiology/superstition is a conduit for unsubstantiated claims, which then allows people to blame, hope and explain an occurrence which hasn't yet been scientifically proven or in some cases has been proven but is being denied because it threatens individual interests.
In Europe, between 1560-1600 more than 8,000 women were killed because of claims that they were witches. Witches were connected/associated with illness, storms, even bad behavior. These associations fueled the perceptions and persecutions, for example it was a "fact" that witches never operated alone, if one was found, then there were others close by. All you needed to do was look for another incident of illness, bad weather, bad crops, etc and there you would certainly find more witches. And, if women spoke out against this practice they were essentially drafting their confessions to the practice of witchcraft.
This may seem a stretch from nutritional epidemiology, but it's really not. When we make a connection based on a narrative that fits our preconceived notions and call it fact, we can easily find "evidence" to further our theory. But none of that makes it true.
This idea that connecting two observations somehow creates a conclusion is immature and unscientific for obvious reasons. Could observations be a starting point in the development of a hypothesis? Of course. Anything more than that is lazy and dangerous.
It is interesting to me that the European witch hunts coincided and then ended around the time scientific tools are developed: the microscope, the telescope, calculus, etc. The scientific tools allowed for methods and discovery that had a better predictive value than the superstitions/epidemiology had had. Reason, after all, is subjective to perspective whereas measurements, standards, etc introduce objectivity without the limitations of emotion. Tools and rigorous standards allow us to understand cause better than superstitions/epidemiology.
Even when we look at smoking, which is often touted as a shining example of epidemiology saving lives, it doesn't tell us anything about the root cause. As Gary Taubes writes in the Case Against Sugar, most cigarettes were soaked in sugar, and those that weren't blended with sugar, like the French Gauloises, didn't seem to cause cancer. Not conclusive, but interesting and completely missed by the epidemiological research at the time. Here's another: About 10% of cigarette smokers get lung cancer, but about 90% of all cases of lung cancer are attributed to (or “caused by”) cigarette smoking. Interesting right? That should provoke further rigorous study...
I loved this line: Meta-analyses become “weighted averages of expert opinions” rather than objective assessments of the data. This is exacerbated by the fact that analyses of a single data set can be split across hundreds or thousands of papers; the Nurses’ Health Study alone has yielded more than 1,000 publications.
Feels like a virus of bad info spreading... (too soon?)
Very few understand this flaw, and those who do would never follow existing guidelines. Epidemiology is garbage.
The Challenge of Reforming Nutritional Epidemiologic Research
3