Sometimes, it doesn’t really matter what the numbers say.
Queer uses for probability theory
That’s the title from a chapter of the book Probability Theory : The Logic of Science by E. T. Jaynes. In this particularly well named chapter, Jaynes is looking at data from an experiment related to psychic phenomena. One particular case involves a woman who seems to have the ability to predict the next card with better than random accuracy. Apparently the chance of getting any one card right was 0.2, and the trial was repeated many thousands of times. If she had no special powers, then the expected number of correct guesses would have been 7420 out of 37100. Instead, she got 9410 guesses correct. Clearly this is more than one would expect by chance alone, even allowing for a few standard deviations of variation (the standard deviation in this case should be about 77, which means she scored an astounding 25 standard deviations above the expected mean, which is, technically speaking, frickin’ impossible for any close to normal distribution). But to put some number to the chances of her getting the results she did by chance alone, Jaynes does some more calculations.
He calculates, using the binomial distribution, the chance of getting exactly 7420 correct answers out of 37100 trials with the probability of success set to 0.2 per trial. So of course, the chances of getting exactly 7420 guesses correct are very small. This gives us the probability for our “null” hypotheses, that: that only pure chance was occurring. Then, the author says, we need to calculate the chance of this event happening if indeed the calmed psychic did have powers to the extent that she had a 0.2538 chance of guessing the correct card (her true observed frequency). He calls each of this chances likelihoods, then goes on to calculate the relative likelihoods, or Lp/Lf where Lp is the likelihood of getting the observed data given our null hypotheses, and Lf is the likelihood of getting the data given that the person really did have enough psychic ability to bump their probability to .2538 chance. When we calculate this ratio, we still get a very small number, which means that, given our data, the “psychic” hypotheses is much more likely than the null hypothesis.
Now, a couple things here. First there’s the method Jaynes used to calculate probability for the null hypothesis in terms of relative likelihoods, and the value of that. But I’m going to set that aside for now (I’ll discuss that topic more in a future blog post). Let’s get to the rub. despite data which would seem to make an extremely strong case for the existence of psychic abilites (at least for this woman), Jaynes argues that the experiment would actually make most scientists less likely to believe in psychic abilities. How can that be, you say? Jaynes takes a kind of Bayesian approach. Basically he says that, when evaluating data and their significance, people take into account their prior beliefs and likelihoods for alternative interpretations. Basically, if you believe strongly before hearing about this experiment that esp is bunk, then when this new data is shown to you it will be weighed against all the other possible explanations. These other explanations (experiment was does wrongly, results were faked, lady found a way to cheat) are then viewed as much more likely than the probability she really is psychic, and so that’s what the scientist ends up believing. The experiment, instead of proving the existence of esp, reinforces the belief that all attempts to show esp have failed.
Did you catch that? Jaynes does a good job of interpreting the implications of this method of belief, in terms of science and paradigm shifts. Some very good stuff here. One final note, Jaynes’ book was written not so many years ago, long after “queer” acquired its current #1 meaning, so kudos to the author for using the term in its powerful, original sense.
