Roger Pielke, Jr quotes from the IPCC’s AR4 report
The uncertainty guidance provided for the Fourth Assessment Report draws, for the first time, a careful distinction between levels of confidence in scientific understanding and the likelihoods of specific results. This allows authors to express high confidence that an event is extremely unlikely (e.g., rolling a dice twice and getting a six both times), as well as high confidence that an event is about as likely as not (e.g., a tossed coin coming up heads). Confidence and likelihood as used here are distinct concepts but are often linked in practice.
Pielke rightly became perplexed by this language. What could it mean? He asked his readers (and me via email) to consider the following:
Here are some specific definitions to help you answer some questions.
A. “high confidence” means “about 8 out of 10 chance of being correct”.
B. “extremely unlikely” means “less than 5% probability” of the event or outcome
C. “as likely as not” means “33 to 66% probability” of the event or outcome
So here are your questions:
1. If the IPCC says of a die that it has — “high confidence that an event is extremely unlikely (e.g., rolling a dice twice and getting a six both times)” — how should a decision maker interpret this statement in terms of the probability of two sixes being rolled on the next two rolls of the die?
I answered this puzzler on Roger’s blog (Roger showed two questions, but they are the same at base), but I thought it worth developing further here.
The answer is that there is no answer; or rather, that there are an infinity of answers. The IPCC’s language of “high confidence that an event is extremely unlikely” is ambiguous and incomplete.
Remind yourself that all probability is conditional on certain, exactly specified information, evidence, or premises. What are the premises or evidence for a “high confidence that an event is extremely unlikely”?
Our evidence specifies that “high confidence” means that a statement has 0.8 chance. Here, we have 0.8 chance of an “extremely unlikely event,” and our evidence specifies that this event (call it A) has probability less than 0.05.
We have 0.2 chance missing. That is, there is an 0.8 chance that A is extremely unlikely. But we need the full probability to say what is the probability of A. This must mean that there is a 0.2 chance that A is something other than extremely unlikely. The IPCC does not specify what this “other” than extremely unlikely is, so it could be anything.
We can provide our own evidence to provide a solution. Suppose, just for fun, that the 0.2 chance is for an event A that is merely unlikely, which we specify to mean 0.1 probability. Then we can write a cartoon equation:
Pr(A | this information) = 0.8 * (Prob < 0.05) + 0.2 * 0.1 = 0.8 * (Prob < 0.05) + 0.02.
And that’s as far as we can go. Whatever 0.8 * (Prob < 0.05) becomes, we add 0.02 to it. The problem is that we do not know what (Prob < 0.05) means. Does it mean “more likely to be 0.05 than 0.01″? Or “equally likely to be any number between 0 and 0.05″ or something else entirely? There is no language in the IPCC that allows us to discern which of these (or some other) is true.
The IPCC’s language is either sloppy thinking or shrewd politics. Given my experience with actual, working scientists, I tend to believe the former. But if it’s shrewd politics, regardless whether A happens or not, the IPCC has given itself wiggle room to say that it predicted A wouldn’t happen, or that it predicted A wasn’t particularly unlikely.
I say this because though we cannot come to an exact solution, we can find its bounds given the language we do have. First, we know there is a 0.8 chance that (Prob < 0.05): the lowest this can be is 0 (just in case (Prob < 0.05) means 100% certainty of 0), and the highest it can be is 0.05 (just in case (Prob < 0.05) means 100% certainty of 0.05). Thus 0.8 * (Prob < 0.05) is between 0 and 0.04.
Now the 0.2 chance. The probabilities available to us are those between 0.05 and 1 (or so it seems; the language is still ambiguous). This means 0.2 times whatever this is is bounded between 0.01 and 0.2.
Our solution is then
Pr(A | our information) in [0.01, 0.24].
Thus, if A did not happen, the IPCC would point to its prediction and say, “See! We told you so. We said A was nearly impossible and it didn’t happen.” But if A did happen, it could say, “Well, A happened, it’s true. But it happens about 1 out of 4 times, which isn’t that unlikely. We can be satisfied with our prediction.”
Full discussion here