Log in / Register for the Open

The Framingham Heart Study, Part 1: Cargo Cult Science

27
ByDr. Michael EadesJanuary 14, 2019

Part 1: In the beginning — Framingham’s Inception

On April 12, 1945, in Warm Springs, Georgia, President Franklin D. Roosevelt spent the morning going over matters of state and meeting with guests, some of whom had “commented on how well he looked.” Shortly after lunch, while sitting for an artist, the president complained of a headache. Two hours later, he was dead from a massive cerebral hemorrhage (1).

The untimely death of the president at 63 set in motion forces that would bring about the most famous study in history: the Framingham Heart Study (FHS) (2). The FHS laid the groundwork for the obsession we’ve had with cholesterol and saturated fat and may well be the study that has been most damaging to the health of the U.S. population. This damage comes not necessarily from the study’s data but from the misreporting, deception, dissembling and outright prevarication about the data that have made it the wellspring for both the diabetes and obesity epidemics that afflict us.

Over time, the messages on diet and cancer have been ratcheted up until they are almost co-equal with the smoking messages. I think a lot of the public is completely unaware that the strength of the message is not matched by the strength of the evidence.

It’s difficult to overestimate the impact this one study has had on the way physicians and patients view the causes and treatments of heart disease. The FHS is revered in most scientific and medical circles as the finest long-term observational study ever performed. The data generated from this ongoing study have been published in over 1,000 scientific papers and were used to create the Framingham Risk Score, a gender-specific algorithm used by physicians around the world to estimate the 10-year risk for cardiovascular disease in patients.

Just over 70 years ago, on Oct. 11, 1948, physicians examined the first Framingham subject and the study officially began (3). Since that day, not only has the original group of subjects been repeatedly examined, but the children and grandchildren of the original cohort have also been examined. In the early 2000s, other cohorts were added to increase diversity; the original cohorts were mainly Caucasian of European origin. Due to the length of the study and prestigious academic credentials of those directing it over the years, the FHS is considered a landmark study—if not the landmark study—on the risks for developing cardiovascular disease. Many researchers hold that the Framingham data strongly support the lipid hypothesis—the notion that cholesterol in the blood leads to the development of heart disease.

Dr. William Castelli, one of its early directors, said this of the FHS:  “It is a place that discovers, proves, establishes in an epistemological sense what are the risk factors for heart disease. The findings of Framingham have already helped millions of people around the world, and even if the older generation is not helped directly, their children, grandchildren and great grandchildren will be helped.” (4)

But not all the researchers who were deeply involved in the process feel the same way. Dr. George Mann, an early Framingham researcher whose name is on many FHS articles, has been outspoken about his lack of regard for the way the National Heart, Lung, and Blood Institute (NHLBI) disregards data that don’t confirm the lipid hypothesis. According to Mann, failure to report the enormous amount of contrary data “is a form of cheating” indulged in frequently by the NHLBI, the government-funded organization that runs the FHS (5).

It can doubtless be said of any large study—especially one such as the FHS—that there will be glitches along with a smattering of malcontents and detractors. If you are a doctor or scientist, the path of least resistance is clearly to go along with the crowd and embrace the findings of the FHS, which has been funded to the tune of millions upon millions of dollars and produced a mountain of papers authored by esteemed investigators from prestigious institutions. But, as everyone knows, the path of least resistance isn’t always the correct path.

In order to make any kind of an intelligent determination about the validity of the FHS, we need to examine what’s really going on.

Cargo Cult Science

In my view, the entire FHS is what Nobel laureate Richard Feynman called “cargo cult science” in a commencement address in 1974 at Caltech. This is a pursuit that is not really science but has all the trappings and the outward appearance of science (6).

When Feynman coined the term, he was referring to a South Sea culture that blossomed financially during World War II. Military cargo planes brought goods and created a booming economy for the island, and when the economy crumbled after the war, the islanders decided to try to get the cargo planes to return. They did so by trying to recreate the situation that prevailed during the war. They “arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for the airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land.”

Feynman calls this kind of effort cargo cult science “because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.”

So what essential quality is missing from the FHS, and why won’t the planes land, so to speak?

How can 1,000-plus papers—many of which are filled with all the differential equations, matrices and statistical analyses that are the “apparent precepts and forms of scientific investigation”—be flawed?

First, the FHS is an observational study, which, by definition, can’t prove causality. Observational studies are valuable for formulating hypotheses that can then be tested by more rigorous means to try to determine causality. An observational study alone won’t do it.

Observational Studies and Causality

The notion that an observational study can’t prove causality is foreign to many people. Intuitively, it seems that causality would be proven if researchers show a correlation between some risk factor and a disease it is thought to cause. But it doesn’t work that way. One of the axioms of science is that correlation is not causation.

To better understand this concept, consider a made-up observational study.

Imagine a doctor who is part of an eight-physician practice back in the early 1950s. He has been seeing patients for 30 years, as have the other doctors in the practice. Our doc notices that he’s been seeing a lot of bronchitis cases, and he hears from his patients who are smokers (as most adults were in the 1950s) that they’ve cut down while they’re sick because cigarettes seem to make their bronchitis worse. Hearing this repeatedly, the doctor wonders if maybe smoking makes people more prone to bronchitis. He rounds up the medical records of patients going back to the 1920s, when the medical practice first formed, and separates them into two stacks: one stack of charts for patients still with the clinic, the other for patients who are no longer in the practice because they died, moved or found another doctor.

He then takes the charts of those still in the practice and separates them into two groups: one group of patients who are smokers and a much smaller pile of those who are not. After scouring these records for episodes of bronchitis he and the other physicians have treated over the past 30 years, he discovers that the clinic’s smoking patients got bronchitis 15 times more often than the nonsmoking patients. Smoking definitely correlates with risk for bronchitis. Knowing the hazards of tobacco, it’s easy to conclude that smoking causes bronchitis. But it can’t be done with this kind of observational study because observational studies can show only correlations; i.e., this risk correlates with that disease. And remember, correlation is not causation.

But it seems so obvious that smoking caused bronchitis. Given the data provided above, how could anyone not come to that conclusion? Because maybe there is another factor we don’t know about—this should always be remembered. Maybe another factor causes both the smoking and the bronchitis.

I come from a family of smokers. Both my parents smoked, my four siblings smoked, my grandparents smoked, all my aunts and uncles smoked, and yet I never had any inclination whatsoever to smoke. My wife has the same history. What makes the two of us different from the rest of our family members?

Going back to the patients in our imaginary study, maybe they harbor some third factor that makes them likely to smoke and prone to bronchitis. Or maybe a propensity to develop bronchitis due to some slight change in the chemistry of lung secretions causes mild symptoms that are relieved by smoking. In the former case, smoking might not cause bronchitis even though the observational evidence strongly points to the idea that it does. In the latter case, the propensity for bronchitis could actually cause smoking.

So how do we prove causality? In theory, you take the hypothesis developed from the observational study and do a randomized controlled trial (RCT), the so-called gold standard of experiments to determine causality.

In our example above, we can start with the hypothesis that smoking causes bronchitis. To prove this, we would need to recruit a number of subjects who are nonsmokers into our trial. Then we would have to randomize them into two groups and have one group start smoking a couple of packs a day while the other group continued to abstain from cigarettes. We would follow these two groups closely to see if the smokers experienced more cases of bronchitis than the nonsmokers. If they did, we could say that smoking causes bronchitis with some certainty—especially if other researchers repeated the study with other subjects and got the same findings.

It should be immediately obvious that this kind study couldn’t be done for ethical reasons. In many situations, researchers are stuck with observational data, such as most of the data on smoking and bronchitis, lung cancer, heart disease, etc.

There are other ways to make evaluations to bolster the issue of causality. Animals can be studied to see if they respond to tobacco by developing bronchitis. The doctors in our observational study above could simply ask their bronchitic patients to quit smoking. Some would, some wouldn’t. If the ones who did quit reduced the rates at which they were afflicted with bronchitis as compared to the ones who continued to smoke, and even as compared to themselves back when they smoked, the hypothesis that smoking causes bronchitis would be strengthened.

Our imaginary observational study shows how science usually works. First, someone makes an observation. In our case, the physician noticed that his patients who were smokers came down with bronchitis more often than his nonsmoking patients. He generated a preliminary hypothesis: Smoking causes bronchitis. He devised a way to test the hypothesis with an observational study using his 30 years of patient data. His preliminary hypothesis held up in that his data showed an enormous increase in bronchitis episodes in his smoking patients.

But what we’ve got to remember is that despite how strongly the data implicates smoking as causing bronchitis, we have only a hypothesis at this point. And, it must be noted that the data in this imaginary observational study is vastly stronger than most data in these kinds of studies. Our smoking/bronchitis study above showed smokers developed bronchitis at 15 times the rate of nonsmokers. That is huge! In most observational studies, the difference in rate between one group and another is maybe 1.2 or even 1.15, which, if translated to our case, would mean that for every nonsmoker who got bronchitis, 1.2 smokers got it. That’s pretty weak gruel, yet those are the findings of most observational studies.

And, by the way, when you read press reports of studies saying, for instance, that people who avoid red meat live longer or people who eat bacon die sooner, you’re almost certainly reading about an observational study. You can always tell by the weasel words used in the article. The authors never say bacon causes early death; they say bacon is “linked” to earlier death or that red meat is “associated” with a shorter life. Or they say that avoiding red meat is “correlated” with increased lifespan. Those words—linked, associated, correlated and a handful of others—are a dead giveaway that the study in question is observational and thus worthless at proving causality.

How can I know that my risk for heart disease has increased because my cholesterol is a little high if no one has proven that cholesterol in the blood causes heart disease?

It’s crucial to understand that these kinds of studies cannot prove causality no matter how meticulously they are performed.

If the FHS is an observational study—which it most assuredly is—how can the results be used to develop a risk-factor scoring system? That would imply someone had proven causality. How can I know that my risk for heart disease has increased because my cholesterol is a little high if no one has proven that cholesterol in the blood causes heart disease?

That’s a good question—and one that remains unanswered by the FHS. We’ll examine this and other flaws and failings of arguably the most famous and widely disseminated observational study ever published in more specific detail in Part 2.


References

  1. Bruenn HG. Clinical notes on the illness and death of President Franklin D. Roosevelt. Annals of Internal Medicine 72(4): 579-591, 1970.
  2. Dawber TR. The Framingham Study: The Epidemiology of Atherosclerotic Disease. Cambridge, Mass.: Harvard University Press, 1980.
  3. Mahmood SS, Levy D, Vasan R et al. The Framingham Heart Study and the epidemiology of cardiovascular disease: a historical perspective. The Lancet 383(9921): 999-1008, 2014. Available here.
  4. Brody JE. Scientist at work: William Castelli; preaching the gospel of healthy hearts. The New York Times. Feb. 8, 1994. Available here.
  5. Mann GV. Coronary Heart Disease: The Dietary Sense and Nonsense. Cambridge, England: Janus Publishing Company, 1993.
  6. Feynman RP. Cargo cult science. Engineering and Science 37(7): 10-13, 1974. Available here.

All links accessed Jan. 14, 2019.

Comments on The Framingham Heart Study, Part 1: Cargo Cult Science

27 Comments

Comment thread URL copied!
Back to 190115
Matthieu Dubreucq
October 25th, 2019 at 12:43 am
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

If you go back to 190113 the video that talks of causality vs observation gives similar le examples on that subject.

I have never really took the time to realize that must study we read in regular media is based on observations.

Comment URL copied!
Katina Thornton
January 17th, 2019 at 10:14 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Dr. Eades, I enjoyed your article. I never took statistics and was not trained in how to critically dissect a journal article, published paper, or a study. I simply took my premed classes, put my head down for the grind of medical school and residency, and trusted the "smart doctors" training me to tell me what was correct and what was incorrect. Therein lies the flaw. I have spent the last three years unlearning much of what I blindly took as the truth. CrossFit, and what is now CrossFit Health, initiated that journey and have been with me every step of the way. The greatest gift is that now my children understand the value of skepticism and careful interpretation of what we read and hear.

Comment URL copied!
Veronique Oomen
January 15th, 2019 at 6:14 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

In order to prove causation, correlation is necessary but not sufficient. You also need a plausible mechanism based on accepted scientific principles. In the absence of a plausible mechanism, ice cream consumption definitely causes sunburn.


In order to take drastic actions like condemning entire food groups such as red meat, we should demand not just a "feel-good" explanation of a mechanism, but proof that every step claimed to take place, actually does. And the latter is where the lipid hypothesis has fallen short time and again.


And maybe it's just me, but lines of thinking that rely on the absence of self-regulating mechanisms, (like calorie-counting as the only way to stay lean; do you really think your body doesn't regulate weight...?) or discount what humans evolved to do and eat (do you really think that survival does not depend on health?), can never make sense. So I'll stick with lifting heavy, constantly varied movement, little or no sugar and enjoy my steak. And when a doc prescribes statins, I'll change doctors.

Comment URL copied!
Paul Rosch
January 15th, 2019 at 5:46 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Dr. George Mann was a co-director of the Framingham study and was involved early on to develop a nutritional survey to evaluate the effect of diet on cholesterol. His extensive analysis of the results completed by 1960 was never published, probably because it found that participants had widely varying cholesterol levels and that "something explains this inter individual variation, but it is not diet." William Kannel, Director of the Framingham study from 1966 to 1979, never referred to this, and Ancel Keys, a strong proponent of a low-fat diet who was referred to in the media as "Mr. Cholesterol", was featured on the cover of the January 13, 1961 issue of Time magazine. As one leading authority triumphantly proclaimed, "No other variable in the mode of life beside the fat calories in the diet is known which shows such a constant relationship to the mortality rate from coronary or degenerative heart disease".


Unfortunately, the Framingham study has had more of an impact on CHD research than any other epidemiological study. A review of the 10 greatest advances in cardiology in the last century listed Framingham in second place with "Lipid Hypotheses" and Atherosclerosis in third. The reason for this is that it allegedly provided the first "solid evidence" that people with a high cholesterol were more likely to have a heart attack and that smoking and hypertension also increased risk. William Kannel, Director of the Framingham study during the 1960’s, told the press that the Framingham results essentially proved that cholesterol was a powerful predictor of CHD. However, this was not supported by data showing that half of heart attacks occurred in people with normal or low cholesterol. A direct association was later reported between falling cholesterol levels over the first 14 years of the study and increased mortality rates over the following 18 years. A 30-year follow-up study in 1987 stated, "The most important overall finding is the emergence of the total cholesterol as a risk factor of CHD in the elderly". No data were presented to support this erroneous claim. Indeed, for men above the age of 47, those with low cholesterol had mortality rates greater than those with high cholesterol. In addition, those whose cholesterol had decreased spontaneously over 30 years were at greater risk of dying from heart disease than those whose cholesterol had increased. "For each 1% mg. drop in cholesterol there was an 11% increase in coronary and total mortality.


Although the study found that a drop in cholesterol was associated with increased coronary deaths, it was cited as supporting the cholesterol-CHD link in The Cholesterol Facts, a subsequent joint American Heart Association-NIH publication, which stated that “The results of the Framingham study indicate that a 1% reduction in cholesterol corresponds to a 2% reduction in CHD risk The real truth about diet and cholesterol came out in a 1992 Archives Of Internal Medicine article stating that, "in Framingham, Mass, the more saturated fat one ate, the more cholesterol one ate, the more calories one ate, the lower the person's serum cholesterol". The author was William Castelli, Director of the Framingham study at the time.

Comment URL copied!
Emily Pisarski
January 15th, 2019 at 4:42 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Interesting article! Experienced researchers should not argue that correlation=causation. You will find that researchers are often careful to use any causal language since as you point out, most findings from any study lead us to a hypothesis not a concrete outcome. Yet, to understand how various behaviors affect our health in the long run, it is challenging to avoid using observational studies. The funding and follow-up necessary to complete a 30 year long RCT is usually just not feasible. Before you hold RCT's as the gold standard, keep in mind that even within RCT's it is often very difficult to make sure that everyone is getting the exact same dose of an intervention. CrossFit Health readers may enjoy reading qualitative research studies as they tell individual stories that once again, provide us with pieces of a puzzle that may someday add up to optimal health outcomes. Qualitative studies can also oftentimes give us insight into what is truly happening to people in various arms of a randomized controlled trial vs. what is supposed to be happening given their study arm assignment.


It is more than fair that CrossFit is questioning research methodologies and academic journal publications, but as part of the CrossFit Health scale-up, I'd urge you to also take a look at the medical education system as a whole. If we can interrupt the cycle of standard practices that are not backed by research and/or not working to improve health, we can have better outcomes here in the US.

Comment URL copied!
John Nisbet
January 15th, 2019 at 3:32 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Love the direction that CrossFit is taking. Started in late 2008 and was initially taken by the fitness and sport. My introduction was taught by Graham Holmberg as it happens. As time has evolved, I'm fitter at 44 than was at 34 - and light years more educated as it relates to health. Have a background myself in physiology, and a close family friend struggling with Type 2 diabetes, so this direction and refocus for content is invigorating to help others such as her.


I'm sure CF Health folks review a LOT of what is current and has been historically published, but I found this and those like it to be very interesting as it relates to helping those with diabetes. Off the couch, off the carbs, so to speak.


https://www.ncbi.nlm.nih.gov/pubmed/29508274


Thanks Coach, for what you and CF are doing.

Comment URL copied!
Uffe Ravnskov
January 15th, 2019 at 2:49 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

I agree with Michael Eades; association is not the same as causation. As said by the philosopher Karl Popper, a medical hypothesis can never be proven, but it can be falsified, and the cholesterol hypothesis has been falsified again and again. Let me for instance point at the fact that more than a dozen studies have shown that people with low cholesterol become just as atherosclerotic as people with high cholesterol. It is also a fact that although cardiovascular disease is the commonest cause of death, elderly people with high LDL-cholesterol live the longest. Read our BMJOpen review of 19 studies where the authors have followed more than 68,000 elderly people after having measured their “bad” LDL-cholesterol. In most of them, those with the highest LDL-cholesterol lived the longest. No study found the opposite: (https://www.ncbi.nlm.nih.gov/pubmed/27292972 ).

This fact was documented already in the 30-year follow-up study from Framingham. Here the authors stated the following about those who had passed the age of 50: ‘For each 1 mg/dl drop in cholesterol per year, there was an eleven percent increase in coronary and total mortality’ (JAMA. 1987;257:2176-80).


The reason why high cholesterol is associated with heart disease in younger people is most likely that young and middle-aged people are more stressed than retired people, and stress is able to raise cholesterol by up to 40% and may cause heart disease by other mechanisms than by raising cholesterol.


If high cholesterol was the cause of atherosclerosis and cardiovascular disease, a high degree of cholesterol-lowering should of course be more beneficial than a low-degree lowering, but this is not the case; read our recent paper in Expert Review of Clinical Pharmacology: https://www.tandfonline.com/doi/pdf/10.1080/17512433.2018.1519391?needAccess=true . Here we have also shown that there are many more falsifications of the cholesterol hypothesis.


As an argument against Michael Eades´ view, Stanley Nasraway mentions that he and his wife have been tremendously financially prosperous at selling books and services on nutrition to the lay public. I doubt that they have earned much money this way. I myself have published three books about the cholesterol scandal; some of them in ten different languages. However, most of the publishers pay very little or nothing at all. For instance, Amazon, who is selling them all over the world, rarely pays me more than €30-40 per month. Those who become prosperous are those who support the cholesterol campaign, but they are clever to hide it. Recently, the American Heart Association published new guidelines for prevention of cardiovascular disease: https://www.semda.org/wp-content/media/New-2018-Lipid-Management-Guidelines.pdf . According to a large table on pages 71-73, none of the authors have any financial conflicts. However, beneath the table you can read the following:


The table does not necessarily reflect relationships with industry at the time of publication. A person is deemed to have a significant interest in a business if the interest represents ownership of ≥5% of the voting stock or share of the business entity, or ownership of ≥$5,000 of the fair market value of the business entity; or if funds received by the person from the business entity exceed 5% of the person’s gross income for the previous year.


May I suggest that Nasraway read my homepage: www.ravnskov.nu or my first book, The Cholesterol Myths (It is freely available in English from Smashwords: https://www.smashwords.com/books/view/486704). To learn more about the criminal ways used by the drug industry and their well-paid supporters, I suggest a reading of my book Ignore the Awkward. How the Cholesterol Myths Are Kept Alive.

Comment URL copied!
Alex Broadbent
January 16th, 2019 at 3:30 am

Hi - just to be clear, Eades doesn't just say that association is not causation, which is surely correct. He also says that observational studies can't prove causation.


In fact, all science can ever establish is associations, no matter what methods you use.


There's then an INDUCTIVE inference (not a logical, deductive one - see Greg Glasman's comment above) to the conclusion that causation is present.


In this regard Dr Eades's piece is (in my opinion) flawed. In particular note that all the evidence for Crossfit, as for any other exercise programme, comes from "observational" research, as opposed to randomized controlled trials. That's because you can give someone a placebo pill and not tell them (this is "blinding") but you can't put someone on an exercise programme and not tell them! (Sadly...)

Comment URL copied!
Alex Broadbent
January 15th, 2019 at 2:21 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Regarding the idea that observational studies can't prove causality, this is not correct, and is also an instance of the sort of crude over-simplification that I think the author would want to resist.


First things first - you can't really "prove" causality; it's an inductive inference.


Secondly, it's an over-simplification; observational studies CAN "prove" (= render probable enough to care seriously about) causality, it's just rare, and you usually need many of them, all triangulating on the same conclusion.


A better statement might be: observational studies always suffer from the risk of residual confounding, and it is very rare that a single study can be considered decisive for a causal conclusion.


Randomised controlled trials can likewise suffer from residual confounding, and suffer a number of problems of their own, one of which is the excessive faith placed in the protocol, which in reality is often breached. (The best piece of evidence for this is that it's much more common for a trial funded by a body with a vested interest in an outcome, such as a pharmaceutical, to find a positive result, than if it's done by an independent.)


Note also that the evidence for smoking causing lung cancer was all observational until very recently (long after everyone accepted that smoking causes lung cancer). The triangulation of results between many different studies, with different designs, performed by different investigators in different places on different subjects etc., can establish causality with sufficient certainty to legislate extensively and sue tobacco companies. See Austin Bradford Hill's legendary 1965 paper, "The Environment and Disease: Association or Causation?"


BUT most importantly for Crosstif - I don't think I exaggerate if I say that all the evidence for Crossfit is from observational studies. This is true for any exercise programme and is the curse of sports science generally. It's impossible to implement blinding protocols for an exercise programme.


Thus I think the CrossFit endorsement of a stance against all observational studies is self-defeating. Strong, clear statements are great, but they're not always true, and unfortunately the truth is sometimes complicated - as indeed is the argument of this piece in relation to Framingham.


Thanks for the provocation and stimulation!

Comment URL copied!
Alex Broadbent
January 15th, 2019 at 2:22 pm

PS I'm an academic who publishes on causal inference in the health sciences, among other things; I don't have any commercial interests in this matter

Comment URL copied!
Greg Glassman
January 15th, 2019 at 2:25 pm

Scientists are careful not to use the word "proof" knowing that there's no such thing, no such need, in science. "Proof" is the exclusive province of mathematics and logic.

Comment URL copied!
Clarke Read
January 15th, 2019 at 5:18 pm

Alex,

I have three questions for you, and I mean them very sincerely because I've asked them to lots of others without getting satisfying answers.


1. Regarding causal inference from observational studies, what relevance do the original Bradford Hill criteria have today? They're the only formal set of guidelines I'm aware of, and what I don't know is whether they still provide guidance to the field today, either directly or implicitly. If there's a more recent set of standards, I'd love to know about them.


2. Beyond the widely-cited example of smoking, are there any other modern (say, past 50 years) examples of successful causal inferences from observational research?


3. One situation that seems to come up frequently in the health sciences is a large number of observational studies all pointing in the same direction, but with weak magnitudes of effect (both individually and when analyzed in summary). How does such a situation shift the bar for causal inference, and at what point are you confident you are not observing the impact of a consistent confounder you have failed to account for?


Frankly, given the enormous costs and logistical constraints related to controlled clinical research, I would love for there to be at least a theoretical path forward for effective observational research in health sciences.

Comment URL copied!
Alex Broadbent
January 15th, 2019 at 6:10 pm

@ Clarke Read - I have a lot of reading I could recommend on this (including some by me... but by others too). There were a lot of these guidelines floating around in Bradford Hill's time. I suspect this was because of the political context: they were fighting corporate interests to establish something, and a long list of "alternative arguments" is what lawyers really need. It's not actually a formal method. More recently, some epidemiologists have pushed for properly formal methods for analysing the results of observational studies. The basic idea is to try to interpret them as if they were trials. I have a couple of papers on this in International J of Epi (I'm a philosopher of science, but my two co-authors are both eminent epidemiologists) and a couple of pure philosophy ones too. I criticise the approach, but also acknowledge its merits. I can point you to some excellent papers by advocates of the approach, too.


If you're serious about looking into this then a good place to start would be the December 2016 special issue of International Journal of Epidemiology, with papers by many major contributors to this ongoing debate.


I would also recommend you look at the work of Judea Pearl - he has a recent "popular" book called The Book of Why - where he tries to make his approach accessible to a broader audience. But it's frankly very difficult - he's an excellent mathematician, but not always easy to follow.


Another excellent person to look at is Nancy Cartwright, a philosopher of science who has written a great deal about this. She has a book with Hardie


Briefly to your question about successful obersvartional research - yes, definitely. For example, sudden infant death syndrome reduced massively when it was (eventually) agreed to alter the advice from front-sleeping to back-sleeping. It is important to distinguish areas. Nutritional epidemiology is in an absolute mess, in my view. It's just so hard to manage residual confounding. People who eat more veg also tend to earn more, have less stress, drink less, smoke less, etc. Similarly for things like exercise, meditation, religious practice... all the lifestyle things we would really like to know about. But there are other areas with much greater success, such as occupational epidemiology (e.g. asbestos). There's a history of epidemiology by Alfredo Morabia that may be of interest to you. Also take a look at the STROBE collaboration, which is an observational answer to the Cochrane collaboration.


This is far too long... happy to liaise further if you like, but perhaps we should take it off comment thread.

Comment URL copied!
Clarke Read
January 15th, 2019 at 9:07 pm

Thank you, Alex. Really appreciate you taking the time to put that together. I similarly appreciate your presentation of the space, which seems as well-reasoned as I could hope for.


I'll start digging into that stuff and get myself acquainted. I'll reach out directly once I've done a bit more homework of my own. You have at least brought me to the level of conscious incompetence, so I have some work to do.


Reading your points now, I think some of us with backgrounds specifically in nutrition and adjacent fields may be unjustifiably generalizing from what we've observed to the field of epidemiology more generally. I'm hopeful the positive examples, methods and discussions you've mentioned can illustrate the standard (quality & quantity) certain inferences require without throwing out any sort of evidence categorically. That could be a much more productive state than many of the discussions in nutritional epidemiology I've seen recently.


Thanks again.

Comment URL copied!
Alex Broadbent
January 16th, 2019 at 3:26 am

Not at all Clarke - you're welcome (and you're not incompetent!).


I did write an academic book on this and can reference several of my own pubs but hesitate to be accused of a profit motive. (If only people knew, though, how little money one can make from an academic book!)

Comment URL copied!
Stephen Dull
January 15th, 2019 at 12:55 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

I hung with this article for a while, but the conclusion was a little far reaching. There does have to be more evidence then just making an observation. There needs to be data, tests and other correlations.


I understand where he wanted to go with this article, but it didn't quite get there.

Comment URL copied!
Greg Glassman
January 15th, 2019 at 12:41 pm
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

All,


I edited Stanley Nasraway's post and I’m reposting it.


First I edited for a string of straight-up factual errors all of the same genus. Everywhere he falsely claims FHS demonstrated that X increases Y, claiming a causal relationship, instead of an increase in X was associated with an increase in Y, claiming only a correlation, I struck through red and then deleted.


Next, I edited for the Ad Hominem attacks. Stanley does a LOT of ducking. Some of it was spurious insult and ad hominem. All that shit I struckthrough yellow and then deleted.


You can’t come in here and use as presumptive evidence the very fault in logic that the author was pointing out and sandwich that crap with ad hominem attack and insult.


Look, everyone, at what’s left when you take away the crap? Still crap, but less of it. It turns out Stanley has absolutely nothing to contribute to the conversation. In, fact, Stanley/Mary are here only to stop the conversation. They won’t last long.


This is what an industry troll looks like:


STANLEY NASRAWAYJanuary 14th, 2019 at 6:16 pm


I wondered where he was going when he states "the study that has been most damaging to the health of the U.S. population". The FHS is an incredible longitudinal study that has enriched our knowledge and pointed organized medicine in many very important directions. Its findings include:


1970s

In women who are postmenopausal, risk of heart disease is increased, compared with women who are premenopausal.

1980s

No empirical evidence found to confirm the rumor that filtered cigarettes lower risk of heart disease as opposed to non-filters.

1990s

Elevated blood pressure can progress to heart failure. Framingham Risk Score is published, and correctly predicts 10-year risk of future coronary heart disease (CHD) events. At 40 years of age, the lifetime risk for CHD is 50% for men and 33% for women.

2000s (high normal blood pressure is called prehypertension in medicine; it is defined as a systolic pressure of 120—139 mm Hg and/or a diastolic pressure of 80—89 mm Hg). Lifetime risk of developing elevated blood pressure is 90%. Obesity is a risk factor for heart failure.


By pointing to smoking, hypertension, hyperlipidemia, it has had an astonishing impact on the reduced mortality we see today from heart disease and stroke. Statins, which decrease hyperlipidemia, have been proven by randomized controlled studies to decrease heart attacks and stroke, and to extend lifespan.

Comment URL copied!
Malcolm Kendrick
January 15th, 2019 at 10:55 am
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Where Michael Eades and I will disagree, in degree, is that I believe correlation can prove causation, if the correlation is strong enough. I think it beyond reasonable doubt that smoking causing lung cancer. There have never been any controlled studies in this area - not will there ever be. However, if memory serves, smokers are seventeen times more likely to develop lung cancer than non-smokers. At this level of correlation we have effectively proven causation.


I have discussed this issue with many statisticians and epidemiologists. Some say that if risk is more than doubled, causation can be accepted. Others say you need the risk to be increased by 5 - 6 fold to accept causation. So, the rules are not hard and fast.


On the other hand if the risk is less than double, you can basically ignore any observational study. I don't think I have seen a single dietary study where risk gets anywhere near doubled. When it comes to cholesterol we are looking at 1.16, and suchlike. This would be laughable, were it not so serious.


For those who say that the statin trials have proven the cholesterol hypothesis, I would say, please review the data with an open mind. For example, evacetrapib (designed to raised HDL and lower LDL) lowered LDL 37%, raised HDL 120% and had no impact on CVD. The benefits of statins are almost certainly due to non-lipid effects.


If the Framingham Study had shown anywhere near a doubling of CVD risk with raised cholesterol it could be considered interesting, but it never did. The only finding that blew the doors of was that, if participants cholesterol levels fell during the first fourteen years of the study, the rate of CVD death increased dramatically during the next eighteen years.


'There is a direct association between falling cholesterol levels over the first fourteen years [of the study] and mortality over the following 18 years (11% overall and 14% CVD death rate increase per 1mg/dl per year drom on cholesterol levels). Journal of the American Medical Association, 24 April 1987, pages 2176 to 2180.'


If cholesterol fell 18%, the risk of death increased by 420% and the risk of CVD death by 546%. So, yes, the Framingham study can tell us some things.

Comment URL copied!
Alex Broadbent
January 16th, 2019 at 3:23 am

Agreed that correlation is causation, but it's important to dispel the idea that the risk must be doubled. This isn't scientific, it originates in a legal context, where corporates were trying to argue that unless RR > 2, causality was not proved and they could not be held liable. (I can give references to some cases if you're interested.) In fact, it's possible for relative risk to be any amount and for causation to be present. Even if an exposure raises risk by less than 2, it's still raising the risk. Heart-related risks are like this, INCLUDING those related to smoking. Thus smoking raises risk of lung cancer by an enormous amount (maybe 17 as you suggest, maybe more) but only raises risk of fatal heart disease by about 1.7. Yet smoking kills many more people via the heart disease route (so to speak) than by the smoking route, because lung cancer is a very rare disease even among smokers. So it's important not to discount exposures which elevate the risk by a relatively small amount - obviously it depends how large the risk is in the first place! I've written on this myself - both the legal uses of the RR > 2 test, and the importance of considering both absolute and relative risks - and so have some leading epidemiologists / biostatisticans, notably Sander Greenland.

Comment URL copied!
Katina Thornton
January 17th, 2019 at 9:49 pm

I loved reading your book "The Cholesterol Con." The humor that you invoke on a topic of such great importance was refreshing. I'm excited that your daughter is going to be a pbysician, and I hope that she keeps her father's torch for the truth lit.

Comment URL copied!
Jeffrey Cain
January 15th, 2019 at 5:27 am
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

Dr. Eades presents an introduction to the Framingham study that explicates the difference between causation and correlation. He asserts that Framingham is an observational study, which limits its findings to the realm of correlation, not causation. Do you take issue with Dr. Eades’ central assertion? Are you asserting that causation can be proven through observational studies? It would be helpful if you engaged the argument instead of attacking the messenger, which is the least useful and lowliest form of comment, as you well know.

Comment URL copied!
Alex Broadbent
January 15th, 2019 at 6:11 pm

Hi Jeffrey - I did in fact engage the argument - and I didn't attack Dr Eades. Yes, observational research can provide evidence for causation. Smoking and lung cancer; exercise and reduction in obesity; crossfit and ...

Comment URL copied!
Alex Broadbent
January 16th, 2019 at 3:17 am

Ah sorry I see this reply wasn't addressed at me... I agree that ad hominem attacks are totally wrong

Comment URL copied!
Stanley Nasraway
January 15th, 2019 at 2:16 am
Commented on: The Framingham Heart Study, Part 1: Cargo Cult Science

I searched Dr. Eades and see that he and his wife have been tremendously financially prosperous at selling books and services on nutrition to the lay public. A PubMed search shows he has never published any research or scholarly article in a recognized medical journal. Knocking the status quo is self-serving and self-congratulatory, similar to celebrity pumpers like Drs. Oz or Mercola. I wondered where he was going when he states "the study that has been most damaging to the health of the U.S. population". The FHS is an incredible longitudinal study that has enriched our knowledge and pointed organized medicine in many very important directions. Its findings include:

1960s

Cigarette smoking increases risk of heart disease. Increased cholesterol and elevated blood pressure increase risk of heart disease. Exercise decreases risk of heart disease, and obesity increases it.

1970s

Elevated blood pressure increases risk of stroke. In women who are postmenopausal, risk of heart disease is increased, compared with women who are premenopausal. Psychosocial factors affect risk of heart disease.

1980s

High levels of HDL cholesterol reduce risk of heart disease. No empirical evidence found to confirm the rumor that filtered cigarettes lower risk of heart disease as opposed to non-filters.

1990s

Having an enlarged left ventricle of the heart (left ventricular hypertrophy) increases risk of stroke. Elevated blood pressure can progress to heart failure. Framingham Risk Score is published, and correctly predicts 10-year risk of future coronary heart disease (CHD) events. At 40 years of age, the lifetime risk for CHD is 50% for men and 33% for women.

2000s

So called "high normal blood pressure" increases risk of cardiovascular disease (high normal blood pressure is called prehypertension in medicine; it is defined as a systolic pressure of 120—139 mm Hg and/or a diastolic pressure of 80—89 mm Hg). Lifetime risk of developing elevated blood pressure is 90%. Obesity is a risk factor for heart failure.


By pointing to smoking, hypertension, hyperlipidemia, it has had an astonishing impact on the reduced mortality we see today from heart disease and stroke. Statins, which decrease hyperlipidemia, have been proven by randomized controlled studies to decrease heart attacks and stroke, and to extend lifespan.


I provide this post simply for balance and perspective, and to identify HYPE when we read it. Crossfit Health unfortunately doesn't provide the reader this perspective, or any characterization of the author, for that matter. That is an unfortunate oversight. It would be intriguing to know if CF recruited this 2 part piece from Dr. Eades, or if Dr. Eades contacted them and volunteered it. Just curious.

Comment URL copied!
Shakha Gillin
January 15th, 2019 at 4:57 am

Dr Eades makes great points about observation studies, correlation and causation.


I do not equate academia and journals with credibility. Unfortunately academia at medical societies have been hijacked by industry funding.


I don’t write papers, I don’t work at a university....but you know what I do....I save lives.


Dr Eades and Dr Eades are highly regarded physicians who I have the utmost for.


Looking forward to Part 2.

Comment URL copied!
Shakha Gillin
January 15th, 2019 at 4:58 am

Edit- Utmost respect for.

Comment URL copied!
Scott Gillin
January 15th, 2019 at 5:16 am

Hi Stanley I find it intersting that you attack Dr. Eades personally rather than addressing the topics in the article (ad hominem?). Do you believe that correlation is the same as causation? Do you believe that more than a hypothesis can be drawn from an observational study? These are very important points to understand about science and it is a disservice to the CrossFit community to try to undermine the credibility of a person you only know through a Google search.


I like that you want to challenge the information being published. It is a vital part of any healthy discussion to look at all points of view. For this to be beneficial the discussion should be focused on the science. Personal slander should have no place on this discussion board.

Comment URL copied!