A defense of classical statistics against the bayesians

by Hans Olav Melberg

Introduction
There are lots of attacks on classical statistics, and only a few brave souls seem wiling to defend it philosophically. So, although my sympathies are Bayesian, I would like try to present a couple of arguments against Bayesian statistics and in favour of the classical approach.

The main argument is practical. When I read an article I am simply not very interested in the priors of the author or any other experts. All I need to update my beliefs and my priors, is the result of the experiment. In addition to this practical defence of the clasical approach, there are also some conceptual problems with the Bayesian method such as the problem of how to update on beliefs with probability one or zero.

Brief definitions
A Bayesian interprets probabilities as subjective beliefs and updates her beliefs using Bayes theorem when she receives new evidence. A classical statistician interprets probabilities as a measure of relative frequency and has little use for Bayes' theorem since they believe the priors are subjective and therefore they should not be used as inputs in order to arrive at a objective and scientific conclusion.

In practice
To examine the practical differences between the two approaches, consider how you would arrive at a probability about the effect of a new drug. The Bayesians will start by forming a prior belief about the effect of the drug, then conduct an experiment, then use the results from the experiment in order to update the prior beliefs and arrive at the final estimate of the effect of the new drug (the posterior distribution). The classical approach is to go straight to the experiment and present the results from this experiments without forming or updating on prior probabilities.

Now, philosophically I have little doubt that the Bayesians are right that probabilities should be interpreted as subjective beliefs. However, this does not mean that I am very interested in the prior beliefs of the author about the consequences of the drug and their probabilities. I have my own priors and all I need from the article is the result from the experiment in order to update my own beliefs. At a practical level therefore, all I want is the information contained in the classical approach.

One might argue that this is a bit too harsh and that it would be interesting to know the priors of a group of experts. I agree! However, although presenting expert opinions is an advantage, I still do not think this should be aggregated to a prior belief and used as an input in an updating process before presenting the final verdict. Why not? Because there is no such thing as a "group subjective belief." Only individuals have beliefs and there are lots of aggregation problems and psychological biases in a group process leading to the "expert consensus." So while I would like to be informed of these beliefs in an article, I do not want the author to use an expert prior, or his interpretation of what the expert consensus is, and update on this. Based on the information available and my reading of the opinions I have to form my own opinion of the initial priors and all I really need and want from a paper is the classical results: the relative frequencies.

What about beliefs with probability one or zero?
Let's say that you make a mistake and that you falsely believe that something is impossible. An experiment then shows that it is possible. Clearly you should change your mind! The problem is that it is impossible to do so if you are a true Bayesian. You simply cannot update beliefs which are assigned probabilities that are zero or one. They stay likie that forever regardless of the evidence.

This is clearly absurd and I guess that most Bayesians would like to shout at me that this simply means that you should not assign something a probability of zero or one. So instead of being a problem, they turn it into a partial virtue: As Bayesians we are careful not to bee too sure because we are aware of the foolishness that would result from being too certain. And the argument that Bayes therem does not work for zeros and one is simply a demonstration if this foolishness.

Although I am attracted to this argument, there are still some problems. First of all, to put it in general terms: What if I happen to be a fool at some point about some issue and assign it a probability of zero or one, I would to have a system that allowed me to recover and not be stuck! To point out that it was stupid of me to do something like that in the first place, does not eliminate the problem that it is impossible to get out. True, I should not have done so in the first place, but what if! That is the problem, and it is a problem the Bayesians cannot escape. (At least this is what I believe with a certainty of about 0.87)

Let me add a second argument that makes thiungs even worse. Stating that you should never assign something a probability of zero or one (unless it is an analytic truth, like a mathematical theorem), is easy to do if the list of possible outtcomes is well defined. However, what if the state space is infinite or there are some states you simply have not thought of? In theory this is equivalent to assigning a alternative a zero probability and in practice this is a much easier mistake than explicitly assigning something a probability of zero or one. We may be fools for being absolutely certain about an outcome we have heard about, but it seems much more common and less foolish simply to fail to list all the possible and relevant outcomes. Logically the two are euivalent for Bayesians. They must assume a complete description of the relevant state-space. This, I think, makes it more difficult to dismiss the zero/one argument as foolish. Instead it becomes a conceptual problem for the Bayesians: How to do statistics in a world in which we often fail to list all the possible outcomes?

Conclusion
I am a Bayesian in the sense that I think beliefs are subjective. However, I am most interested in the result of experiments when they are untainted by other people's subjective beliefs. In this way it becmes easier for me to update my own beliefs using my own priors. So in this sense I like the Bayesian philosophy, but I still like articles to have a classical approach. However, even the Bayesian philosophy givens me some worries sine it seems to lead to the absurd conclusion that we shouold maintain beliefs that are obviously false given that we at some point made a foolish mistake or, slightly less foolish, made a mistake when imagening the possible outomes.

Literature
There is a ton of literature on this topic, mostly by Bayesians pounding on the Classical approach. A philosophical discussion from both perspectives can be found in "Bayes or Bust." An interesting and more formal attack on Bayesianism is "The Limitation of Bayesianism" by Pei Wang.