Ideas19 Dec 2007 06:47 pm

UPDATED

Wide open mouth

Some topics in probability and statistics.

- Variance
- Combinatorics
- Expected value
- Conditional expectation
- Marginal probability
- Joint discrete and continuous distributions
- The characteristics of the Gamma, Beta, Normal, Cauchy, Exponential, T, F, Uniform, and Chi-square continuous distributions
- The characteristics of the Geometric, Binomial, Poisson, and Uniform discrete distributions
- Moment generating functions
- Probability generating functions
- Convergence in probability, convergence in distribution
- The Central Limit Theorem
- The Weak and Strong Laws of Large Numbers
- Independence
- Bayes theorem
- The theorem of total probability
- De Morgan’s laws
- Change of variable transformations
- Cumulative Distribution Functions (CDF) and Probability Density Functions (PDF)
- Probability Mass Functions (PMF)
- Order statistics
- Indicator functions
- Various mathematical series (Taylor, Geometric)
- The Jacobian
- Markov’s inequality
- Chebyshev’s inequality
- Covariance
- Correlation
- Disjoint sets
- Runs tests

Ideas30 Nov 2007 10:15 pm

Sometimes it seems like an event is so rare, so incredible, that there’s only one explanation. But consider the alternative universes.

The Anthropic Principle

Cosmologist (not cosmetologist!)

There’s a detailed, complex argument behind the Anthropic Principle, which in its limited, original version justifies the existence of life against apparently infinitesimal small scientific odds of a life-sustaining universe. But I think the most important part of the Anthropic Principle can be summed up very easily:

“If the outcome had been negative, we wouldn’t be around to witness it.”

In other words, if the universe had been inhospitable to human life, no one would be around to verify that the most probable outcome (in the absence of of a “Creator”) came to pass. But the principle itself goes well beyond the cosmological. It effects how we understand any rare event which happens to happen to us.

Start with an extreme event. A fully loaded bus slides over the guardrail on a mountain pass. Ninety nine passengers die. One survives. It would hardly be surprising if that one passenger says God, and not random luck, saved her. After all, she alone survived, miraculously, while everyone else perished in a wreck which seemed destined to kill everyone.

But how do the other 99 passengers feel? Did God choose them to die, just like He apparently choose her to live? Maybe we could interview them. Oh, wait.

Take thousands of individual events, each one extraordinary improbable, and try them out with billions of people over and over. The most likely result, the most scientifically probable, is that some people will experience extremely unlikely occurrences. If they go on to ascribe those experiences to more than just blind luck that shouldn’t surprise us. It’s only natural. But it most certainly doesn’t prove divine intervention.

Think of it this way: You, the consciousness reading these very words, are the product of millions of little events which could have just as easily gone the other way. If you mother didn’t have a soft spot for men with mustaches… If you grandfather hadn’t been shot down over France… If you great-great grandfather hadn’t bumped his head on that giant bottle of Sam’s Cure-all Tonic… you won’t be here. You’re as unlikely as a tossed coin landing on its side.

Of course the same thing goes for everyone else around you. Ten billion souls, each one coming into existence despite near impossible odds. Ten billion miracles, right? Only if you discount the ten heptillion souls who’s coin flips landed the usual way and were never born. They are mute, silent, YouTube-impaired, invisible non-witnesses to their own bell-curve filling banality.

Ideas29 Nov 2007 05:09 pm

Sometimes, it doesn’t really matter what the numbers say.

Queer uses for probability theory

Queer eye for estimating probabilitiesThat’s the title from a chapter of the book Probability Theory : The Logic of Science by E. T. Jaynes. In this particularly well named chapter, Jaynes is looking at data from an experiment related to psychic phenomena. One particular case involves a woman who seems to have the ability to predict the next card with better than random accuracy. Apparently the chance of getting any one card right was 0.2, and the trial was repeated many thousands of times. If she had no special powers, then the expected number of correct guesses would have been 7420 out of 37100. Instead, she got 9410 guesses correct. Clearly this is more than one would expect by chance alone, even allowing for a few standard deviations of variation (the standard deviation in this case should be about 77, which means she scored an astounding 25 standard deviations above the expected mean, which is, technically speaking, frickin’ impossible for any close to normal distribution). But to put some number to the chances of her getting the results she did by chance alone, Jaynes does some more calculations.

He calculates, using the binomial distribution, the chance of getting exactly 7420 correct answers out of 37100 trials with the probability of success set to 0.2 per trial. So of course, the chances of getting exactly 7420 guesses correct are very small. This gives us the probability for our “null” hypotheses, that: that only pure chance was occurring. Then, the author says, we need to calculate the chance of this event happening if indeed the calmed psychic did have powers to the extent that she had a 0.2538 chance of guessing the correct card (her true observed frequency). He calls each of this chances likelihoods, then goes on to calculate the relative likelihoods, or Lp/Lf where Lp is the likelihood of getting the observed data given our null hypotheses, and Lf is the likelihood of getting the data given that the person really did have enough psychic ability to bump their probability to .2538 chance. When we calculate this ratio, we still get a very small number, which means that, given our data, the “psychic” hypotheses is much more likely than the null hypothesis.

Now, a couple things here. First there’s the method Jaynes used to calculate probability for the null hypothesis in terms of relative likelihoods, and the value of that. But I’m going to set that aside for now (I’ll discuss that topic more in a future blog post). Let’s get to the rub. despite data which would seem to make an extremely strong case for the existence of psychic abilites (at least for this woman), Jaynes argues that the experiment would actually make most scientists less likely to believe in psychic abilities. How can that be, you say? Jaynes takes a kind of Bayesian approach. Basically he says that, when evaluating data and their significance, people take into account their prior beliefs and likelihoods for alternative interpretations. Basically, if you believe strongly before hearing about this experiment that esp is bunk, then when this new data is shown to you it will be weighed against all the other possible explanations. These other explanations (experiment was does wrongly, results were faked, lady found a way to cheat) are then viewed as much more likely than the probability she really is psychic, and so that’s what the scientist ends up believing. The experiment, instead of proving the existence of esp, reinforces the belief that all attempts to show esp have failed.

Did you catch that? Jaynes does a good job of interpreting the implications of this method of belief, in terms of science and paradigm shifts. Some very good stuff here. One final note, Jaynes’ book was written not so many years ago, long after “queer” acquired its current #1 meaning, so kudos to the author for using the term in its powerful, original sense.

Ideas27 Nov 2007 09:52 pm

Demetrius George Synodinos, aka Jimmy the Greek, oddsmaker and Ohio native

I have my second Statistics exam tomorrow at the U of T. I’d like to say that I’m confidently prepared, but I’m not. I’m still having a hard time with change of variable for joint distributions, applying the Jacobian, and keeping track of all the interactions, rules, and formula for marginal density, conditional probability, and conditional expectation.

To help me out I wrote out all the relevant equations on a single sheet of paper, to study from. It looks like Greek. Really, I mean that literally. Sigma, lambda, omega. The language of mathematical notation is Greek. Even if I put in a mediocre performance on the test (by the way, here in Canada students don’t “take” exams, we “write” them), the fact that I can looking at my page of dense notation without my head exploding means something. Quite a lot, actually.

One of the biggest changes in understanding the math behind statistics is that so much of the notation is confusing or worse. Check out this equation from the Wikipedia article on Joint distribution:

Joint distribution equation

That brief equation manages to pack in 8 x’s and y’s. Four of each. Some are upper case, some are lowercase, some are separated by a comma, one pair has parenthesis around it and another doesn’t. Maybe this really is the best way to express the idea that the double integral of the joint probability density function for x and y, evaluated over the full range of each, is equal to 1. But I doubt it.

At this point the notation is fixed, so I suppose there’s not much use complaining about it. I merely note that one of the “barriers to entry” for learning statistics is a notation which borders on the criminally inscrutable.

Ideas26 Nov 2007 05:38 pm

that this blog will take? I give it one-to-one. In other words, even money. The topic, certainly, is something I can’t escape. For the next few years, perhaps more, statistics and probability will be my everyday bread and butter. My study, and, most certainly, my livelihood. They already are to a great extent. Perhaps I should explain this, but instead with my first post I’d like to discuss one of my favorite ideas in statistics. It’s called the St. Petersburg Paradox.

The St. Petersburg Paradox

Daniel Bernouli, probability god

Here’s the idea. A fair wager is one in which the amount you expect to win is equal to the amount you pay to play. The amount you expect to win, your expectation, is the average (mean) amount you would win over time, if the gamble were repeated over and over ad infinitum. For example, in the case of a single coin toss, if you were to get $2 for guessing right, a fair wager would be $1 since you will guess correctly one-half of the time.

Now suppose that you were offered the following bet: You toss a coin until it comes up heads. If it lands on heads once before tails comes up, you win $2. For each additional time tails comes up before heads, your winnings are doubled. In other words, two tails wins you $4. Three tails wins you $8, four tails wins you $16 and so on. How much should you be willing to pay to play this game? Turns out the expectation for this game is infinite. In other words, if you played this game an infinite number of times, your average winnings would themselves be infinite.

Here’s how that happens. To calculate expected value we multiply the value of each possible outcome by the probability of that outcome. The chance of getting a return of $0 is 1/2, the chance of getting a return of $2 is 1/4, the chance of getting a return of $4 is 1/8 and so on. Put them all together and the sum looks something like this:

St Petersburg Paradox


 

 

As you can see the expectation adds up to an infinite amount! So how much would you be wtilling to pay to play this game? To make it fair on the casino, you would have to ante up everything you had, and then everything else too, just to play once. But the reality is that half of the time you play, you lose your entire bet. And even when you win something, the vast majority of the time it is very little (less than $1000). You expectation may be infinite, but your reality is one of zero or modest winnings most of the time. So your rational willingness to bet is much lower than your expected winnings.

The St. Petersburg Paradox, as it’s come to be know, was described by Daniel Bernoulli in a 1738 article published in Commentaries of the Imperial Academy of Science of Saint Petersburg.