As predictions are the ultimate end of scientific models, validation (or the lack thereof) of a model’s prediction(s) is the ultimate end of scientific experiments and will serve as the focus of this final post in our consideration of the seven elements of science.
Recalling previous discussions, we reiterate that scientific models created from real data or by reasoning from real data must admit validation. This validation confirms or verifies the model’s predictions of qualitatively new phenomena or relationships.
We also recall that prediction is proposed as the ultimate objective for science, which must do better than chance to a prescribed error level.
Validation, then, is the ultimate process of demonstrating the predictions of scientific models. It consists of devising experiments, gathering confirming or denying data, and evaluating the model in question. The predictive value of a model includes a measure of accuracy in the prediction (though not the utility of the prediction).
When the prediction introduces a novel cause-and-effect relationship, a confirming experiment introduces a degree of validation. This is confirmation but of a prediction. The model, while not yet certain and destined never to be certain, is stronger. Facts exist where previously there were none. An instance of validation often becomes part of the domain of the model in a larger context. If the data simply reproduce data already in the domain, they contribute to the accuracy estimate of the supporting data. If the data are unique and still fit the model, the domain of the model increases.
Creativity and Validation
We recall that in our organization of the scientific method, the element of “experiments” was split between the categories of “creativity” and “validation.”
Constructing a validating experiment often requires interpretation, skill, and creativity on the part of the scientist. Science reserves its greatest rewards for the scientist who conceives of an especially clever experiment that either validates a new theory of breadth or disproves a well-established one.
So, does validation begin with the invention of the experiment, or does the experiment belong with the creative process? Even making the prediction or finding it in the model can be a consequence of experimental design. Often the initial formulation of a model does not include what later develops as a major consequence or prediction. Quite often, the predictions of the model are consequences discovered by scientists other than the creator of the model. Placing experimental design under creativity reinforces the objective of emphasizing the creative element of science. The data acquisition phases of experimentation can stay with validation.
Revisiting Model Grading
Validation is an essential element of moving a particular scientific model along the progression of conjecture, hypothesis, theory, and law. For example, a theory is a hypothesis that has at least one instance of validation. One fact has met all the premises of the hypothesis and confirms the prediction of the model. Put another way, this instance of validation, discovered via experimentation, has moved the model from hypothesis (a model based on existing data, making a non-trivial prediction that is as yet unvalidated) to theory.
This chart (taken from Jeff Glassman’s 2019 article “Rating Scientific Models”) provides a helpful visual of validation’s role in rating or grading scientific models.
A Note on Validation
It should be noted that often validation is not as straightforward as these discussions might indicate. It is not as simple as measuring the degree of bending of starlight around the sun. As science advances, scientists create models of phenomena more and more deeply embedded in noise. As the easy models become part of the history of science, the new models encompass more uncontrollable phenomena that can mask a direct observation of the prediction. This is particularly true in epidemiological studies. The experimental design problem in assessing these phenomena challenges the creative skill of the scientist more than developing the original model.
Many models never reach fruition as a theory because of the conceptual problems in designing a method for validation. Nonetheless, many marginal models manage to affect public policy and national well-being far beyond their predictive power. The impact these flimsy, conjectural models are having is a measure of the prevailing low level of science literacy in America and the world at large.
Comments on Elements of Science: Experiments & Validation
Very interesting piece. It’s hugely relevant given the impact unvalidated models have had on our lives recently.
This method of devising a model, testing it with experiments and either increasing the accuracy or range of its predictions seems obvious when you hear it. In fact, so obvious you’d expect post-modern academics to get it entirely wrong. And they did!
Skepticism of the unobserved goes back to Hume. He did not trust inductive reasoning— any contingent proposition about an unobserved phenomenon inferred from an observed phenomenon was not logically sound. This argument is comprehensively evaluated in ch. 4 of Scientific Irrationalism by David Stove.
Popper and his followers took Hume’s inductive skepticism even further and argued that there can be no such thing as “proof” of a theory. To prove a theory would require an infinite number of confirmatory experiments, whereas it only takes one disconfirmatory experiment to disprove a theory. This ultimately leads them to conclude that any experiment or piece of evidence does nothing to strengthen the validity of a theory, since infinitely more evidence is always required.
The power of this piece is that it moves the discussion away from scientific knowledge as true/false statements requiring proof in the mathematical or logical sense, but towards probabilities. Science makes predictions that are reliable to a known degree of accuracy. Further experiments can improve the accuracy or range of predictions.
Psychologist Karl Friston has proposed that this is how the brain works in what he calls the “free energy principle”. Our neural circuitry is a prediction engine. As we go through life, our brains are constantly making predictions about the environment. This becomes a mental model of how the world works. Our actions and observations confirm and disconfirm this model, improving its accuracy over time. The ultimate goal of a healthy brain is to minimize prediction error. In this state, surprise and uncertainty are kept to a minimum.
The free energy principle actually has implications for fitness. We have all experienced workouts that looked tough on paper, but we survived. And of course, the workout was easier the second time through. However, a bad experience can do the opposite. If someone has hurt their back deadlifting, they will come back to it with fear. The expectation of pain is often worse than the pain itself. Some people even develop real sensations of injury, when there is nothing physically wrong with them. Dr. John Sarno wrote a whole book on this. There are people out there with herniated discs in their spine blissfully unaware of it and feeling no pain. Other people feel debilitating back pain even when x-rays and MRIs show nothing physically wrong. I’m sure Dr. Friston would have something to say about that.
The free energy principle has been touted as the biggest theoretical breakthrough in science since Darwin’s theory of evolution. I don’t know about that, but I do see parallels between its ideas and the methods of science presented here.
If the goal of a single brain is to minimize prediction error, the goal of science is to minimize prediction error accross an entire civilization. It's a shared entity consisting of the seven elements: definitions, observations, measurements, models, predictions, experiments and validation. Without science, each brain pursues knowledge on its own, with no hope for growth or accumulation over generations, as you see in most species of the animal kingdom.
Thank you for this insightful exploration of scientific reasoning, theories of knowledge, and their application to the human brain, particularly as envisioned by Karl Friston's free energy principle.
Indeed, the scientific method we use today is built on an understanding of our limited grasp of absolute truth. Recognizing this, we try to progressively refine our models of the world through empirical testing and adjustment, leading to ever more accurate, though still imperfect, representations of reality. This continuous refinement and adaptation is similar to how our brains develop their models of our surroundings, trying to reduce uncertainty and improve predictability.
Like the single human brain, the scientific endeavor can be seen as a civilization-wide "prediction engine." Each piece of scientific research is a hypothesis being tested, a prediction being made, which then gets confirmed or rejected based on experimental evidence. This collective enterprise enables us to gradually build a shared knowledge base that is more comprehensive and accurate than what any individual could achieve alone.
As you pointed out, this method has important implications for fitness and health as understanding the mechanisms behind pain and injury can lead to more effective treatments and preventive measures. It also provides insight into the human condition itself, exploring why we think, feel, and behave as we do.
This is a great series, well organized and presented. Thank you for distilling these complex concepts into something far more easily delivered to the general public.
Elements of Science: Experiments & Validation
3