The Failure to Measure Dietary Intake Engendered a Fictional Discourse on Diet-Disease Relations

Editor’s Note: The authors of this editorial have previously worked with and received funding from Coca-Cola, ILSI, and other industry or industry-funded groups. Their arguments, both here and elsewhere, have been used to undermine the value of nutrition research categorically and to dismiss any link between diet and disease. 

As shown repeatedly on CrossFit.com, however, there is substantial evidence that certain dietary factors contribute to the progression of metabolic diseases and that the removal of certain dietary factors can slow or reverse disease progression. It is thus important to note that while these authors’ arguments do substantially undermine the value of nutritional epidemiology, specifically, as well as other forms of research in which surveys are used to estimate what and how much subjects eat, they in no way reduce the value of the many randomized controlled trials that clearly demonstrate the links between diet and disease. 

Question
Do dietary surveys — used in nutritional epidemiology to assess what and how much subjects eat — provide an accurate estimate of food intake? If not, to what extent are they flawed, and what are the implications of these flaws?
Takeaway
Dietary surveys are both practically and theoretically flawed to an extent that renders them fundamentally unscientific. Any data or conclusions that rely on surveys to assess dietary intake are unreliable and should not be used to inform nutritional recommendations or individual behavior. Given that the majority of public health recommendations regarding the link between diet and disease are heavily informed by nutritional epidemiology, and nearly all nutrition epidemiology has used dietary surveys, the majority of these recommendations are unscientific. The authors argue nutrition research can only regain its credibility by discarding any data or beliefs informed by dietary surveys and exclusively using more rigorous methodologies for all future research.

This 2018 review argues the tool used to estimate intake in the majority of nutritional research — memory-based dietary assessment methods, or M-BMs (1) — is so theoretically and practically flawed that any data or conclusions derived from it are fundamentally inadmissible as evidence.

The authors note the historical successes of nutrition research. More than a century ago, deficiency-related diseases (beriberi, rickets, goiter, pellagra, etc.) were relatively common; today, less than 20% of the population is at risk of a consequential dietary deficiency (2). The effort to identify and mitigate deficiencies in the American diet is a notable public health success. Beginning in the mid-20th century, nutrition research began instead to focus on the link between diet and other noncommunicable diseases, such as obesity, diabetes, and heart disease (3). This research did not directly measure what subjects ate and drank, instead asking them what they recalled eating and drinking (4) — as Walter Willet describes such methods in his seminal textbook on nutrition epidemiology, they represent “perceptions of usual intake” (5). These recalled values were then correlated to risk of disease. The conclusions, which linked specific dietary factors to increased risk of specific disease states, implied that the subjects’ recollections of the foods they ate and drank were accurate reflections of their actual intake.

As early as the 1950s, research indicated these recollections did not provide reliable estimates (6). These findings, along with subsequent research, have repeatedly demonstrated that what subjects say they eat in dietary surveys bears a trivial relationship to the types and amounts of foods they actually consume, and the results of dietary surveys, if taken at face value, often imply diets that are biologically implausible (7). Research has also found the correlations between diet and disease derived from these studies are consistently unreliable; one recent analysis found zero out of 50 claims from nutritional epidemiology were successfully replicated in subsequent randomized controlled trials, with five studies showing correlations in the opposite direction (8).

Despite decades of data undermining their validity, these methods remain prominent: 80% of studies in the USDA’s Agriculture National Evidence Library — the basis for the quinquennial dietary guidelines — are reliant on dietary recall methods, and the National Academy of Sciences specifically argues for their continued use (9). Additionally, data from NHANES, a large regular survey performed by the CDC, uses dietary recall to estimate Americans’ intake levels. This survey is regularly used to inform both public policy and public perception of the healthfulness or unhealthfulness of certain foods (10). When the authors of this review previously argued against the validity of this data in a published editorial, they were met with ad hominem attacks and vitriolic rebuttals (11).

Here, the authors argue indirect dietary assessments are both practically and theoretically flawed to an extent that undermines all data and conclusions drawn from them. Broadly, such assessments provide only uncorroborated self-reported data — a set of anecdotes describing perceived or recalled dietary intake rather than a direct measurement of what subjects ate. They constitute a form of evidence that would be considered clearly inadmissible in many other fields of study (12).

Specific practical issues with such assessments include:

• Deception: It is well-established that when subjects are asked about what they ate, they will deliberately lie, systematically biasing their reported intake toward foods that are considered healthy and away from those considered unhealthy. This error alone, the authors argue, establishes dietary recall as “pure fiction” (13).
• Reactivity: When subjects know their intake is being monitored or that they will have to report on their intake in the future, their dietary patterns shift toward foods they perceive as healthy and away from foods they perceive as unhealthy. More health-conscious subjects who have healthier non-dietary behaviors are more likely to make such shifts, which may exaggerate healthy-user biases. Subjects’ perceptions of what is healthy or unhealthy are themselves often derived from the results of previous research using dietary recall, which then leads to a pattern in which repeated manifestation of bias can be interpreted as support for a dietary hypothesis (14).
• Credulousness: It is unequivocal that even in the absence of deception or reactivity, we are simply unable to accurately recall what we ate. One editorial, speaking about the use of survey-based methodologies across a variety of fields, argued half of what informants report is “probably incorrect.” Nutritional epidemiology, however, analyzes this data as if it provides a completely accurate assessment of each subject’s historical intake (15).

These practical issues are sufficient to disqualify the use of any research using dietary recall surveys to track intake. Additional theoretical issues undermine the methodology at a more fundamental level:

• Categorical errors: Nutrition epidemiology does not measure dietary intake. Instead, it measures subjects’ perceptions of their dietary intake. It is a categorical error to conclude on the basis of any dietary recall that there is a link (or lack thereof) between any dietary factor and a disease state; instead, we can only say that there is a correlation between the recollection of having consumed a certain amount or type of food and the frequency of a particular condition.
• Misplaced concreteness: Nutrition epidemiology treats the dietary recollections subjects provide as direct, concrete measurements of actual intake. They are not. To mistake abstract estimates of intake for direct measurements is pseudoscientific and misleading (16).
• Pseudo-quantification: These reports do not directly measure dietary intake but must derive numeric estimates of the number and types of calories consumed to perform the intended statistical analyses. These surveys attempt to quantify intake values by asking subjects how much of a certain type of food they recall eating or how often they eat this food. Such methods do not produce truly quantitative data but merely a set of numerical values assigned to anecdotal data. The actual dietary intake patterns remain unknown (17).

These errors are compounded by these surveys’ reliance on incomplete and faulty nutrition databases. The USDA database, which many of these surveys reference to turn an estimated amount of food consumed (e.g., 2 cups of broccoli per week) into nutrient estimates (e.g., 62 calories, 60 mg sodium, etc.), contains 8,000 items. The U.S. food supply contains an estimated 85,000 items. Each of these 85,000 items can vary in nutrition content based on production, transport, storage, processing, and preparation, all of which vary over time. These figures clearly indicate that even if these dietary surveys could accurately track the types and amounts of foods subjects ate — and as shown above, they cannot — they would fail to accurately estimate nutrient consumption (18).

Recently developed tools used to monitor dietary intake, such as using photographs of meals to track what subjects eat, are subject to these same issues and so are equally flawed (19).

Crucially, we cannot accurately estimate the direction or magnitude of the errors involved in these dietary assessments. Science requires that we be able to distinguish fact from fiction and accurately assess error. Dietary recalls are “non-falsifiable anecdotes” rather than direct measures of food intake. Thus, dietary assessment data are fundamentally unscientific, as are claims in some papers that these errors can be corrected for (20).

The authors of this review argue these issues are sufficiently ubiquitous and egregious to render spurious the majority of the perceived relationships between diet and disease (21). See Editor’s Note above for additional context.

Notes

1. Note: Two common examples of these dietary recall methodologies are 24-hour recalls (24HR) and food frequency questionnaires (FFQs), which ask subjects to estimate how much and how often they eat particular foods.
2. Sailors’ scurvy before and after James Lind — A reassessment; Pellagra in the United States: A historical perspective; Reemerging nutritional rickets: A historical perspective; CDC second national report on biochemical indicators of diet and nutrition in the U.S. population
3. Diet and epidemiology of coronary heart disease; Epidemiological studies related to coronary heart disease: Characteristics of men aged 40-59 in seven countries; Dietary goals — A progressive view; Serum-cholesterol response to dietary cholesterol: A re-evaluation; The diet and 15-year death rate in the seven countries study 
4. The inadmissibility of what we eat in America and NHANES dietary data in nutrition and obesity research and the scientific formulation of national dietary guidelines; In defense of sugar: A critique of diet-centrism; Controversy and debate: Memory-based dietary assessment; Controversy and debate: Memory-based methods paper 1: The fatal flaws of food frequency questionnaires and other memory-based dietary assessment methods
5. Nutritional Epidemiology
6. Food intakes of obese and non-obese women; The management of obesity
7. Validity of U.S. nutritional surveillance: National health and nutrition examination survey caloric energy intake data 1971-2010; Reply to LS Freedman et al.; A discussion of the refutation of memory-based dietary assessment methods (M-BMs): The rhetorical defense of pseudoscientific and inadmissible evidence; Implausible results in human nutrition research; Critical evaluation of energy intake data using fundamental principles of energy physiology: 1. Derivation of cut-off limits to identify under-recording; Discrepancy between self-reported and actual caloric intake and exercise in obese subjects; Dietary under-reporting: What people say about recording their food intake; Honest but invalid, what subjects say about recording their food intake; Undereating and underrecording of habnitual food intake in obese men: Selective underreporting of fat intake; Do we eat less fat, or just report so?; Lack of efficacy of a food-frequency questionnaire in assessing dietary macronutrient intakes in subjects consuming diets of known composition; Evaluation of under- and overreporting of energy intake in the 24-hour diet recalls in the European Prospective Investigation into Cancer and Nutrition (EPIC); Validity of the assessment of dietary intake: problems of misreporting; Selective underreporting of energy intake in women: magnitude, determinants, and effect of training; Intentional mis-reporting of food consumption and its relationship with body mass index and psychological scores in women; OPEN about obesity: Recovery biomarkers, dietary reporting errors and BMI; Assessing dietary intake: Who, what and why of under-reporting; Energy intake misreporting among children and adolescents: A literature review; Measuring the difference between actual and reported food intakes in the context of energy balance under laboratory conditions; Difference between 24-h diet recall and urine excretion for assessing population sodium and potassium intake in adults aged 18-39 y; Under-reporting of dietary energy intake in five populations of the African diaspora; Relatively severe misreporting of sodium, potassium, and protein intake among female dietitians compared with non dietitians
8. Clinical trials and observational studies to assess the chronic disease benefits and risks of multivitamin-multimineral supplements; Dietary fiber and whole-grain consumption in relation to colorectal cancer in the NIH-AARP diet and health study; Persistence of contradicted claims in the literature; Beta-carotene supplementation and cancer risk: A systematic review and metaanalysis of randomized controlled trials; Concordance Between the Findings of Epidemiological Studies and Randomized Trials in Nutrition: An Empirical Evaluation and Citation Analysis; Deming, data and observational studies
9. USDA National Evidence LIbrary; The disease-mongering of the American diet; NASEM redesigning the process for establishing the dietary guidelines for Americans
10. Implausible data, false memories and the status quo in dietary assessment; DGAC Scientific Report of the 2015 Dietary Guidelines Advisory Committee
11. Controversy and debate: Memory-based methods paper (see entire series of comments)
12. Addressing current criticism regarding the value of self-report dietary data; The problem of informant accuracy; The validity of retrospective data; The seven sins of memory; Our changeable memories: Legal and practical implications; The cognitive neuroscience of memory distortion; Memory and law: What can cognitive neuroscience contribute?
13. What foods do people habitually eat? A dilemma for nutrition, an enigma for psychology; Is the PURE study ‘pure’ fiction?
14. Considering the value of dietary assessment data in informing nutrition-related health policy; The validity of self-reported dietary intake data: Focus on the “what we eat in America” component of the national health and nutrition examination survey research initiative; Understanding nutritional epidemiology and its role in policy
15. The distinctions of false and fuzzy memories; Constructive memory: Past and future
16. The consequences of false memories for food preferences and choices; Science and the modern world
17. The thirteen books of the elements; Measurement in psychology: A critical history of a methodological concept; Measurement: A beginner’s guide
18. The use of implausible data without caveats is misleading; The NHANES dietary data are physiologically implausible and inadmissible as scientific evidence; The impact of revising fats and oils data in the US Food and Nutrient Database for Dietary Studies.; Complexities and opportunities in monitoring and evaluating US and global changes by the food industry; Food composition database development for between country comparisons; Comparison of nutrients in the food composition tables available in the nine European countries participating in EPIC; Separating fact from artifact in changes in nutrient intake over time.; Changes in USDA food composition data for 43 garden crops, 1950 to 1999; Impact of the revision of a nutrient database on the validity of a self-administered food frequency questionnaire (FFQ); Accuracy of stated energy contents of restaurant foods; Monitoring the sodium content of restaurant foods: public health challenges and opportunities
19. Usability and acceptability of the “bite counter” device; Validity of the Remote Food Photography Method (RFPM) for estimating energy and nutrient intake in near real-time
20. Association between dietary factors and mortality from heart disease, stroke, and type 2 diabetes in the United States; Dietary intake among US adults, 1999-2012
21. The contributions of ‘diet’, ‘genes’, and physical activity to the etiology of obesity: contrary evidence and consilience; Cell-specific “competition for calories” drives asymmetric nutrient-energy partitioning, obesity, and metabolic diseases in human and non-human animals; Perspective: Limiting dependence on nonrandomized studies and improving randomized trials in human nutrition research: Why and how.; Bias in the evaluation of low-magnitude associations: An empirical perspective