.:Statistics Blog:.

Finished my final final for the semester last night. The big one: STA257: Probability and Statistics I. Must say that the course was not what I expected. Some good (the professor), some bad (the textbook), some horrific (the T.A.). Below is a list, unordered, of the topics we covered. Many of these I knew about, but we covered them with a mathematical rigor which was truly impressive (and difficult for me to keep up with).

- Variance
- Combinatorics
- Expected value
- Conditional expectation
- Marginal probability
- Joint discrete and continuous distributions
- The characteristics of the Gamma, Beta, Normal, Cauchy, Exponential, T, F, Uniform, and Chi-square continuous distributions
- The characteristics of the Geometric, Binomial, Poisson, and Uniform discrete distributions
- Moment generating functions
- Probability generating functions
- Convergence in probability, convergence in distribution
- The Central Limit Theorem
- The Weak and Strong Laws of Large Numbers
- Independence
- Bayes theorem
- The theorem of total probability
- De Morgan’s laws
- Change of variable transformations
- Cumulative Distribution Functions (CDF) and Probability Density Functions (PDF)
- Probability Mass Functions (PMF)
- Order statistics
- Indicator functions
- Various mathematical series (Taylor, Geometric)
- The Jacobian
- Markov’s inequality
- Chebyshev’s inequality
- Covariance
- Correlation
- Disjoint sets
Quite a list, no? There are probably about a dozen more topics I’ve missed. And all that in just one semester, with just one 2.5 hour lecture per week, plus a worthless hour of tutorial (mostly spent taking quizzes, with a smattering of problem solving from a completely useless T.A. (barely speaks English, makes lots of errors, doesn’t explain what she is doing). Before the final I filled out 6 full pages of “cheat sheet” notes, mostly just formulas. Even for my Linear Algebra course, which seemed very dense, I was only able to fill two pages with formulas to memorize.

Despite the long list, there are of course many items NOT covered. The method at the U of T seems to be to teach these mathematical basics of probability first (with a big emphasis on the particular distributions). Next semester we will be reviewing the many topics related to “Statistical Inference”. These are actually what many people think of as statistics: How good is your estimate? What can you say about a distribution given the data? Regression and bias. The stuff of opinion polls and SPSS.

I should state directly that for me statistics is not, and will not be, about what most people think a statistician does (and what many statisticians in fact do). I’m not studying to become a poll-taker, a social-scientist, a demographer or an actuary (think life insurance). I would imagine that what I am really up to will become clearer as the blog progress. Stay tuned…

My team had a floorball game last night, and once again we won. I helped out with one goal, one assist, and some decent play overall. It’s still early in my first season, so I’m happy to be doing as well as I am, even if I’m not yet able to zorro. I’m also very happy to be in the Elite league and on such a good team. If not for a scheduling conflict, I might have decided to join the (much less competitive) Recreational league for my first year. I also dodged a bullet in the Elite league draft, though for now I’ll leave the specifics unmentioned. Hum, dodging a bullet. What are the odds…

The Anthropic Principle

There’s a detailed, complex argument behind the Anthropic Principle, which in its limited, original version justifies the existence of life against apparently infinitesimal small scientific odds of a life-sustaining universe. But I think the most important part of the Anthropic Principle can be summed up very easily:

“If the outcome had been negative, we wouldn’t be around to witness it.”

In other words, if the universe had been inhospitable to human life, no one would be around to verify that the most probable outcome (in the absence of of a “Creator”) came to pass. But the principle itself goes well beyond the cosmological. It effects how we understand any rare event which happens to happen to us.

Start with an extreme event. A fully loaded bus slides over the guardrail on a mountain pass. Ninety nine passengers die. One survives. It would hardly be surprising if that one passenger says God, and not random luck, saved her. After all, she alone survived, miraculously, while everyone else perished in a wreck which seemed destined to kill everyone.

But how do the other 99 passengers feel? Did God choose them to die, just like He apparently choose her to live? Maybe we could interview them. Oh, wait.

Take thousands of individual events, each one extraordinary improbable, and try them out with billions of people over and over. The most likely result, the most scientifically probable, is that some people will experience extremely unlikely occurrences. If they go on to ascribe those experiences to more than just blind luck that shouldn’t surprise us. It’s only natural. But it most certainly doesn’t prove divine intervention.

Think of it this way: You, the consciousness reading these very words, are the product of millions of little events which could have just as easily gone the other way. If you mother didn’t have a soft spot for men with mustaches… If you grandfather hadn’t been shot down over France… If you great-great grandfather hadn’t bumped his head on that giant bottle of Sam’s Cure-all Tonic… you won’t be here. You’re as unlikely as a tossed coin landing on its side.

Of course the same thing goes for everyone else around you. Ten billion souls, each one coming into existence despite near impossible odds. Ten billion miracles, right? Only if you discount the ten heptillion souls who’s coin flips landed the usual way and were never born. They are mute, silent, YouTube-impaired, invisible non-witnesses to their own bell-curve filling banality.

Had my exam. Went OK. Not great, not awful, just OK. I suppose “OK” is just fine so far as marks go, since my grades (well, anything north of failure) are mostly irreverent. I do want to make sure I am really understanding the material, down to the core, especially at the beginning here of my rigorous training.

But enough about that, I’d like to talk about…

Queer uses for probability theory

That’s the title from a chapter of the book Probability Theory : The Logic of Science by E. T. Jaynes. In this particularly well named chapter, Jaynes is looking at data from an experiment related to psychic phenomena. One particular case involves a woman who seems to have the ability to predict the next card with better than random accuracy. Apparently the chance of getting any one card right was 0.2, and the trial was repeated many thousands of times. If she had no special powers, then the expected number of correct guesses would have been 7420 out of 37100. Instead, she got 9410 guesses correct. Clearly this is more than one would expect by chance alone, even allowing for a few standard deviations of variation (the standard deviation in this case should be about 77, which means she scored an astounding 25 standard deviations above the expected mean, which is, technically speaking, frickin’ impossible for any close to normal distribution). But to put some number to the chances of her getting the results she did by chance alone, Jaynes does some more calculations.

He calculates, using the binomial distribution, the chance of getting exactly 7420 correct answers out of 37100 trials with the probability of success set to 0.2 per trial. So of course, the chances of getting exactly 7420 guesses correct are very small. This gives us the probability for our “null” hypotheses, that: that only pure chance was occurring. Then, the author says, we need to calculate the chance of this event happening if indeed the calmed psychic did have powers to the extent that she had a 0.2538 chance of guessing the correct card (her true observed frequency). He calls each of this chances likelihoods, then goes on to calculate the relative likelihoods, or Lp/Lf where Lp is the likelihood of getting the observed data given our null hypotheses, and Lf is the likelihood of getting the data given that the person really did have enough psychic ability to bump their probability to .2538 chance. When we calculate this ratio, we still get a very small number, which means that, given our data, the “psychic” hypotheses is much more likely than the null hypothesis.

Now, a couple things here. First there’s the method Jaynes used to calculate probability for the null hypothesis in terms of relative likelihoods, and the value of that. But I’m going to set that aside for now (I’ll discuss that topic more in a future blog post). Let’s get to the rub. despite data which would seem to make an extremely strong case for the existence of psychic abilites (at least for this woman), Jaynes argues that the experiment would actually make most scientists less likely to believe in psychic abilities. How can that be, you say? Jaynes takes a kind of Bayesian approach. Basically he says that, when evaluating data and their significance, people take into account their prior beliefs and likelihoods for alternative interpretations. Basically, if you believe strongly before hearing about this experiment that esp is bunk, then when this new data is shown to you it will be weighed against all the other possible explanations. These other explanations (experiment was does wrongly, results were faked, lady found a way to cheat) are then viewed as much more likely than the probability she really is psychic, and so that’s what the scientist ends up believing. The experiment, instead of proving the existence of esp, reinforces the belief that all attempts to show esp have failed.

Did you catch that? Jaynes does a good job of interpreting the implications of this method of belief, in terms of science and paradigm shifts. Some very good stuff here. One final note, Jaynes’ book was written not so many years ago, long after “queer” acquired its current #1 meaning, so kudos to the author for using the term in its powerful, original sense.

I have my second Statistics exam tomorrow at the U of T. I’d like to say that I’m confidently prepared, but I’m not. I’m still having a hard time with change of variable for joint distributions, applying the Jacobian, and keeping track of all the interactions, rules, and formula for marginal density, conditional probability, and conditional expectation.

To help me out I wrote out all the relevant equations on a single sheet of paper, to study from. It looks like Greek. Really, I mean that literally. Sigma, lambda, omega. The language of mathematical notation is Greek. Even if I put in a mediocre performance on the test (by the way, here in Canada students don’t “take” exams, we “write” them), the fact that I can looking at my page of dense notation without my head exploding means something. Quite a lot, actually.

One of the biggest changes in understanding the math behind statistics is that so much of the notation is confusing or worse. Check out this equation from the Wikipedia article on Joint distribution:

That brief equation manages to pack in 8 x’s and y’s. Four of each. Some are upper case, some are lowercase, some are separated by a comma, one pair has parenthesis around it and another doesn’t. Maybe this really is the best way to express the idea that the double integral of the joint probability density function for x and y, evaluated over the full range of each, is equal to 1. But I doubt it.

At this point the notation is fixed, so I suppose there’s not much use complaining about it. I merely note that one of the “barriers to entry” for learning statistics is a notation which borders on the criminally inscrutable.

that this blog will take? I give it one-to-one. In other words, even money. The topic, certainly, is something I can’t escape. For the next few years, perhaps more, statistics and probability will be my everyday bread and butter. My study, and, most certainly, my livelihood. They already are to a great extent. Perhaps I should explain this, but instead with my first post I’d like to discuss one of my favorite ideas in statistics. It’s called the St. Petersburg Paradox.

The St. Petersburg Paradox

Here’s the idea. A fair wager is one in which the amount you expect to win is equal to the amount you pay to play. The amount you expect to win, your expectation, is the average (mean) amount you would win over time, if the gamble were repeated over and over ad infinitum. For example, in the case of a single coin toss, if you were to get $2 for guessing right, a fair wager would be $1 since you will guess correctly one-half of the time.

Now suppose that you were offered the following bet: You toss a coin until it comes up heads. If it lands on heads once before tails comes up, you win $2. For each additional time tails comes up before heads, your winnings are doubled. In other words, two tails wins you $4. Three tails wins you $8, four tails wins you $16 and so on. How much should you be willing to pay to play this game? Turns out the expectation for this game is infinite. In other words, if you played this game an infinite number of times, your average winnings would themselves be infinite.

Here’s how that happens. To calculate expected value we multiply the value of each possible outcome by the probability of that outcome. The chance of getting a return of $0 is 1/2, the chance of getting a return of $2 is 1/4, the chance of getting a return of $4 is 1/8 and so on. Put them all together and the sum looks something like this:

As you can see the expectation adds up to an infinite amount! So how much would you be wtilling to pay to play this game? To make it fair on the casino, you would have to ante up everything you had, and then everything else too, just to play once. But the reality is that half of the time you play, you lose your entire bet. And even when you win something, the vast majority of the time it is very little (less than $1000). You expectation may be infinite, but your reality is one of zero or modest winnings most of the time. So your rational willingness to bet is much lower than your expected winnings.

The St. Petersburg Paradox, as it’s come to be know, was described by Daniel Bernoulli in a 1738 article published in Commentaries of the Imperial Academy of Science of Saint Petersburg.