Meet Paul, forecasting expert

A new method of forecasting has been brought to my attention: Paul. Paul is an English-born octopus, currently living in Germany, and has been predicting with high accuracy the results of the German soccer team, by picking between two boxes marked with the flag of the two competing teams:

How unlikely is it that Paul would have such a string of correct forecasts? Pretty unlikely. If you assume that Paul’s picks were completely random, with a 50% chance of correctly calling the winner, the probability of making 11 good calls out of 12 comes down to 0.29%. Does this mean Paul is the next big thing in forecasting? It’s possible, but I don’t think so (this said with all due respect to Paul and his skills). Leonard Mlodinow, in his excellent book, The Drunkard’s Walk, makes the following comment:

This example illustrates an important point: even with data significant at, say, the 3 percent level, if you test 100 nonpsychic people for psychic abilities […], you ought to expect a few people to show up as psychic.

In other words, if a phenomenon is random, you should typically see the “average” case regularly, but you should also see highly unlikely cases happen from time to time – observing such a rare occurrence doesn’t contradict the randomness of the phenomenon. Or, in the words of the French poet Mallarmé, Un Coup de Dés Jamais N’Abolira Le Hasard (A throw of the Dice will Never Abolish Chance).