The world’s leading publication for data science, AI, and ML professionals.

The myths of modelling: Falsification

We do not verify models by repeated attempts to falsify them, nor should we try; with causality and Bayesian probability we can do better

Photo by Gary Chan on Unsplash
Photo by Gary Chan on Unsplash

The myth

The myth of Falsification has two versions

  • Science progresses through repeated attempts to falsify theories, conjectures or hypotheses. This is the descriptive myth.
  • Science ought to progress through repeated attempts to falsify theories, conjectures or hypotheses. This is the normative myth.

The normative myth of falsification is half of a widely preached doctrine of scientific practice together with the positivist myth of the primacy and objectivity of data— the idea that data are the starting point for all analysis and that we ensure objectivity in our models by gathering data impartially and "letting the data speak".

According to this double doctrine, once the data have "spoken", we proceed to test the hypotheses they prophesied through a process of repeated attempts to prove those hypotheses false. Only thus can we iterate asymptotically towards objective truth.

This belief is remarkably widespread, considering positivism was all but dead as a philosophical movement by the 60s and falsificationism, as we shall see, was effectively refuted 20 years before it was even introduced. But the excesses of postmodern, social constructivism seems to have frightened scientific practice back into a narrow inductive box.

This article and my previous article on positivism argue that this doctrine is at best misguided and at worst directly damaging. Certainly, the practice of modern statistics in certain fields, with a compulsive pre-occupation with hypothesis testing and a hysterical antipathy to explanatory causal models, has atrophied under its influence. But by debunking these myths and letting causality and probability theory take their natural, rightful place in scientific practice, it may not be too late to ensure the practice of data analytics and machine learning in these fields don’t suffer the same fate.

The origins of falsification

Karl Popper (LSE Library - https://www.flickr.com/photos/lselibrary/3833724834/in/set-72157623156680255/)
Karl Popper (LSE Library – https://www.flickr.com/photos/lselibrary/3833724834/in/set-72157623156680255/)

The two myths are unlikely collaborators, as falsification’s great popularizer is none other than positivism’s vengeful nemesis, Karl Popper.

Popper presents falsification as a solution to Hume’s problem of induction. Hume’s problem with induction is that the only basis we have for believing that repeated observation verifies our beliefs is the repeated observation that our verified beliefs have been repeatedly observed. That is to say, the problem of induction is that our belief in induction is inductive.

Popper argues that though we can’t inductively confirm our beliefs, we can deductively show them to be wrong, by the simple observation of deviations from their predicted consequences.

He suggests then that science should then proceed a process of what he calls conjecture and refutation.

The way in which knowledge progresses, and especially our scientific knowledge, is by unjustified (and unjustifiable) anticipations, by guesses, by tentative solutions to our problems, by conjectures. These conjectures are controlled by criticism; that is, by attempted refutations, which include severely critical tests. They may survive these tests; but they can never be positively justified: they can neither be established as certainly true nor even as ‘probable’ (in the sense of the probability calculus)

For Popper, it is in the adjudication between theories we strive for objectivity and impartiality, which he aims to achieve through an adversarial process between protagonists for these theories.

Bold ideas, unjustified anticipations, and speculative thought, are our only means for interpreting nature: our only organon, our only instrument, for grasping her. And we must hazard them to win our prize. Those among us who are unwilling to expose their ideas to the hazard of refutation do not take part in the scientific game.

Why we continue to hold the myth

Popper doesn’t so much bin the tin can of induction as just kick it further down the street, temporarily trading inductive doubt for deductive contingency. Moreover, as we shall see, it turns out after all not to be so easy to refute hypotheses or to design critical tests.

But the process of conjecture and refutation has immense practical, and frankly moral, appeal. By disposing of the notion that data "speak" – that they generate meaning or furnish theories by themselves – and by moving the scrutiny of objectivity away from data and over to a contest between rival theories, Popper’s programme liberates us to conjecture creatively and puts hypothesis back in its rightful place, in a dialogue with data and not subservient to it. Data suggest hypotheses, but hypotheses lead us back into data and, better yet, suggest the search for new data in the attempted refutations of our hypotheses.

Why falsification is a myth

Unfortunately, science no more proceeds by deductive falsification than by naïve induction, nor can it, because we can never test a hypothesis in isolation from the cloud of additional, auxiliary hypotheses that explain how the hypothesis to be tested is connected to the observations that are supposed to refute it. This argument against the possibility of Popper’s "severely critical test" was very elegantly put forward by Pierre Duhem in his "The Aim and Structure of Physical Theory" (in 1906, when Popper was four years old).

In one of his many examples, Duhem discusses aberrations in the orbit of the planet Uranus observed in the mid 1800s. Far from in any way undermining Newton’s universal law of gravitational attraction from which the predicted orbits are derived, these deviations were only ever assumed to challenge an auxiliary assumption – namely that Uranus was not influenced by any other massive nearby object. This, and some of the most heroic hand-calculation in the history of western mathematics, eventually led to the discovery of the planet Neptune. Science progresses thanks to the failure of falsification.

Duhem’s example is particularly poignant as aberrations in the orbit of the planet Mercury were at the time driving attempts to discover a planet whose existence was so assured that it already had a name: Vulcan. It was more than a decade after Aim and Structure was published that Einstein showed that in Mercury’s case, the problem was indeed with Newton’s theory. But Newton’s theory was never seriously challenged by any of its failures until the aberrations in Mercury’s orbit were explained by General Relativity – a theory born of inspirational conjecture if ever there was one.

Duhem’s argument against the reality or possibility of falsification in scientific practice became known as the Duhem-Quine problem after Quine’s contribution to the discussion about half a century later. But Quine’s claim is much stronger.

Quine argues, more in an excess of poetic inspiration than from any particularly well-documented rational process, that any theory can be made to accommodate empirical observation. This is a much more dubious and frankly dangerous claim, because if theory can always be adjusted to fit observation then there are really no constraints on the construction of theory. That way lies madness, or at least social constructivism. We won’t go there. Duhem will do for us.

Why it’s a problem

The only real downside, apart from the cognitive dissonance of preaching a practice that is neither practiced nor even possible to practice, is that with the tools we now have to hand, we can do better than constrain ourselves to falsification in the contest between rival conjectures.

What should we do instead?

Causality and probability are the missing pieces of the Popperian puzzle, because they together provide a natural language for explanatory conjecture with a built-in framework for the broad use of data – confirmatory as well as contradictory – to inform the likelihood of conjectures in an open contest.

Haunted by Hume, Popper is cool towards causality, and the probabilistic framework he would need to assess causal models in the crucible of hypothetical conjecture was in its infancy. Moreover, the prevailing probabilistic framework of frequentism was already the great friend of inductivism (and thus no friend to Popper).

Later readings of Hume suggest that Hume saw the lack of logical basis for believing in causality to be a problem for logic, rather than for a problem for causality, and he was willing to accept that there are real causes in nature, but that our knowledge of them was necessarily contingent and hypothetical.

In this spirit, given the inherently contingent nature of knowledge in Popper’s programme, insistence on the deductive certainty of refutation seems unsupportable. Released from that constraint, we can bring the full arsenal of inferential machinery to bear on a vigorous contest between causal, explanatory models of the problems we have to solve and the data we have to guide us in doing so.


Related Articles