A new book is out by the journalist Tom Chivers, author of The Rationalist’s Guide to the Galaxy and How to Read Numbers. The Wall Street Journal likes it. Kirkus Reviews calls it “An ingenious introduction to the mathematics of rational thinking.” Oliver Burkeman wrote, “Life is shot through with uncertainty, but in this fascinating, witty and perspective-shifting book, Tom Chivers shows why this needn’t condemn us to powerlessness and panic.”
The book is entitled Everything Is Predictable. Spoiler alert: it’s not.
Oh, everything is predictable in the sense that, say, you can shoot at everything in the universe with a Nerf rifle. That, of course, does not mean that you can HIT, say, the planet Saturn with a sponge dart shot from, say, Akron. Just as you can “predict” anything you want, but there is a huge sphere of uncertainty for which prediction can never actually give you useful guidance.
Everything Is Predictable is a paean to the joys of Bayesian forecasting. Bayesian forecasting is what Nate Silver of Five Thirty Eight used to predict election results from 2008 and 2020. And for a while it worked well for him. He forecast the winner of 49 out of 50 states correctly in 2008, in the election between John McCain and Barack Obama. (Indiana went for Obama; Silver had it going for McCain by a slight margin.) In 2012, Silver used Bayesian forecasting again and got 50 out of 50 states right.
Then came 2016. Nate still got 45 out of 50 states right. But the ones he got wrong were Wisconsin, Michigan, Pennsylvania, North Carolina, and Florida, collectively worth 90 electoral votes. Nate did better than almost every other forecaster: he had given Trump a 29% chance to win the election, which is the equivalent of a .290 baseball hitter getting a single, which happens all the time.
But it turns out that the people who eagerly consumed Nate’s forecasts actually weren’t sophisticated Bayesian statisticians. They were mostly educated (but statistically illiterate) liberals who had come to see a Silver forecast as a guarantee that outcomes they preferred were definitely going to happen, and outcomes they feared would not. Nate thought prediction was about uncertainty. His audience thought it was about certainty.
Worse, those eager consumers of polling and forecasts had ceased considering what the two quite plausible alternative outcomes of that election (Hillary Clinton winning, or Donald Trump winning) might actually mean, aside from “relief” and “awful things.” Same goes for those who were watching the polls hoping for Trump to pull an upset. Hillary winning would be depressing to them; Trump pulling out an upset, however, would bring absolute elation, in large part because “establishment elitist” people like Nate had allegedly predicted it could not happen.
Meanwhile, seemingly nobody was actually doing hard thinking about the actual ramifications of a Trump victory. Perhaps least of all Trump. Michael Lewis documents in his book The Fifth Risk how almost nonexistent the Trump transition effort turned out to be. Of course, on the other side, the Clinton folks were doing plenty of earnest work on their expected transition, work that came to nothing at all. And the public at large, lulled by Bayesian forecasters into thinking a Clinton victory was in the bag, certainly was not doing any hard thinking about either outcome. After all, Clinton winning would mean continuation of a Democratic White House, which they probably thought they understood; a Trump win was not supposed to happen, so why think hard about that? So the mainstream media gave them what they seemed to want, which was wall-to-wall Donald Trump rallies, stories about Hillary’s home email system (which appears to be the only one the Russian government did NOT successfully hack into), and above all – POLLS POLLS POLLS. It was all horse race, zero stories about the actual future stakes. The Cult of Prediction, with scandals and sensationalist swirls on top.
The 2016 election outcome was one of the many shocks the Cult of Prediction brought us. They can protest all they want that we all should have read the fine print and known better, but predictors’ success in dominating the national attention space has set us up like bowling pins for shock after shock.
We scenario-planning types, by contrast, are not engaged in prediction at all. We are engaged in identifying and describing multiple plausible alternative future outcomes, and helping our clients think deeply about them and devise strategies to address them. We do this for several reasons.
First, we are interested not in “forecasting,” but in strategic decisions. If you are not going to absolutely rule out some strategic choices based on a prediction, then prediction is not helping your decision process.
In fact, things that are actually predictable – astronomical events; physical, chemical, and some biological processes; increasingly the weather; actuarial outcomes for large populations; and to some degree demographics – tend not to be strategic. Predictability is a quality of phenomena that are well-understood by almost everyone, and hence not a source of competitive advantage to anyone.
Chivers says the following near the beginning of this book:
“Life isn’t chess, a game of perfect information, one that in theory can be ‘solved.’ It’s poker, a game where you’re trying to make the best decisions using the limited information you have…. When we make decisions about things that are uncertain – which we do all the time – the extent to which we are doing that well is described by Bayes’ theorem. Any decision-making process, anything that, however imperfectly, tries to manipulate the world in order to achieve some goal, whether that’s a bacterium seeking higher glucose concentrations, genes trying to pass copies of themselves through generations, or governments trying to achieve economic growth: if it’s doing a good job, it’s being Bayesian.”
But life is NOT poker. Unless it’s a form of poker in which when you lay down three cards, the dealer hands you a lobster, a spatula, and a pine-tree-shaped car air freshener, and the card table gets turned over or flooded on a regular basis. Knowing the odds of a poker game is a lot less useful than anticipating that things like that could happen, and bringing a raincoat and a lobster pot.
And governments failing to achieve economic growth have quite often employed Bayesian approaches; conversely, governments succeeding in achieving economic growth have done so by abandoning prediction and trying things for which there is zero track record – i.e., using rigorous imagination.
Chivers writes something else that actually might be optimistic for humanity:
“Artificial intelligence is essentially applied Bayes. It is, at its most basic level, trying to predict things.”
When you try to predict things, using Bayesian approaches, you are starting with “priors,” that is, your preliminary estimate of the probability of a certain outcome. And that requires some estimate based on the past – crudely, “data.” And as new data – also from the (more recent) past – comes in, you alter your “priors.”
And yet all data is about the past – NOT THE FUTURE. Any predictive system assuming that the past is a reliable guide to the future will work just fine – until the lobsters and floods happen, and it suddenly does not work anymore. Artificial intelligence, as presently constituted, has only the past to work with – worse, it only has data that is already contained within the Internet. Human beings, however, have imagination, which can conjure up entirely new circumstances and events that have never happened before.
And rigorous imagination – the approach we use to writing scenarios of the future for clients – is not imprisoned within that jail of past on-line data. Nor is it concerned with probabilities, at all.
Any Bayesian approach, any probabilistic predictive method, by contrast, cedes the single most important advantage that humans possess over machines: imagination. Prediction is purely derivative of past data. It’s a mathematical function. It assumes you already know what’s important. It assumes the form of the predictive equation is not going to change.
Decades of experience have shown us that that assumption is fatally flawed. In fact, perhaps the only things we really need to know about the future are the things that will change the very form of that “equation,” or render even any accurate results completely irrelevant.
Example of the latter: IBM in the 1980s developed a wonderfully accurate model of the total computing power that would exist from year to year through to the end of the century. But that model failed to imagine, or anticipate, that 99+% of the computers in existence by century’s end would be personal computers. Which would have been better for IBM to have taken on board in 1981: the precise, objectively accurate Bayesian forecast of total computing power; or the revolutionary rise of personal computing, the prologue to the smartphone era?
Like many previous sacred texts of the Cult of Prediction, Chivers’ book cites the work of Philip Tetlock’s and Dan Gardner’s Superforecasters. This group of people, members of the “Good Judgment Project,” have been selected by Tetlock for their ability to predict using Bayesian forecasting. They are asked all sorts of questions about (often) global political and security matters. The questions are carefully framed to admit of a limited number of answers that can be objectively graded, and each has a deadline after which the “correctness” of their answers can be judged. It is important to note that their estimates, in line with Bayesian practice, can be updated right up to the deadline.
The U.S. Intelligence Advanced Research Projects Agency (IARPA) had the “Superforecasters” predicting things such as: “Will Serbia be officially granted European Union candidacy by 31 December 2011?” and “Will the London Gold Market Fixing price of gold (USD per ounce) exceed $1,850 on 30 September 2011?”
“If [superforecaster] Bill Flack were asked whether, in the next twelve months, there would be an armed clash between China and Vietnam over some border dispute, he wouldn’t immediately delve into the particulars of that border dispute and the current state of China-Vietnam relations. He would instead look at how often there have been armed clashes in the past. ‘Say we get hostile conduct between China and Vietnam every five years,’ Bill says. ‘I’ll use a five-year recurrence model to predict the future.’ In any given year, then, the outside view would suggest to Bill there is a 20% chance of a clash. Having established that, Bill would look at the situation today and adjust that number up or down.” – Superforecasting
Tetlock and Gardner write: ““No one bases decisions on Bill Flack’s forecasts…. And that’s unfortunate. Because Bill Flack is a remarkable forecaster.”
We would simply ask Prof. Tetlock and IARPA, “What decisions could possibly be made as a result of Bill’s predictions?” And our answer, quite frankly, would be: “None.”
Of what use is a non-zero or non-100%-certain prediction of China-Vietnam armed clashes to actual decision-makers? They will still have to plan for the “improbable” outcome, won’t they? Yes, they will. Wouldn’t they be far better off dispensing with prediction altogether, and spending their time imagining the ramifications of alternative scenarios of conflict or its lack? OF COURSE THEY WOULD.
What decision could have possibly hinged on whether Serbia was granted EU candidacy by 12/31/11? NONE! As it happens, they were granted candidacy on 1 March 2012. That candidacy remains open but not acted upon. Was the United States going to take any decision for which Serbia’s EU candidacy happening in late 2011 as opposed to early 2012 was going to be the critical criterion? Of course not!
What decision could have depended upon the London price of gold on 30 September 2011? Unless you were a gold trader, which, as a taxpayer, I hope the IARPA is not, the answer is again: NONE! It peaked at $1780.65 in September, then dropped for four solid years. Was the U.S. intelligence community going to go whole-hog and bet on a higher (or even a lower) price for gold as a result of these predictions? NO! Why would they?
Each of the “Superforecasting” questions, like every form of Bayesian forecasting, is very specific and objectively measurable. This makes them very gradable. It also makes them useless for decision-making. Whyis the price of gold above or below $1850? Is a lack of warfare between Vietnam and China proof that they have entered a long-term détente, or will there be a clash a month after the deadline date? Did the lack of Serbian candidacy prior to 31 December 2011 alter its prospects for ultimate membership in the EU? Bayesian prediction can’t help us with these issues, which really are the deeper strategic questions leaders should be grappling with.
Moreover, Bayesian prediction is constantly altered up to the deadline, which does not help with long term strategy at all. It’s like you made a giant bet back in 1992 on the Cleveland Indians to win the American League East in 2024, and you have a Bayesian telling you every day the percentage chance they are going to win the title, decades after you’ve plunked down your money. It’s useless and annoying, and not only because it’s been at 0.00% since 1994. The decision was taken in 1992; it’s now 2024, and it’s far too late to change your bet. What you really needed was an appreciation, back when your decision was made, that Cleveland might no longer even be a member of the American League East, and also that the team might not even exist under that name, as their offensive team name would be changed to the Guardians.
When we were doing scenario planning for the Coast Guard in 1999, they had to pull the trigger on shipbuilding decisions for vessels they were going to have to live with for forty or fifty years. If we had taken a Bayesian approach, how welcome do you think we would have been coming back to them on a daily basis through the 2020s, saying, “The odds that the National Security Cutter program was a good idea have declined by 1.4% today after a 2.3% rise yesterday?”
What decision-makers need is not some constantly swiveling single point prediction – no matter how objective or accurate, by some arcane statistical perspective. They need a sophisticated understanding of the full array of plausible outcomes that they could face, so they can be prepared for all of them that have significant impact on their world of work.
So why is IARPA wasting taxpayer dollars on “Superforecasters?” Because numbers just feel so rigorous and cool. The IARPA people wanted numbers, even though numbers could not possibly help them. They, and most other quantitatively-minded managers, will spend money, time, and staff resources on numbers even when those numbers are of no use whatsoever, and will not spend a fraction of the resources on a rigorous approach to imagining the full range of plausible outcomes that their organization could face in the future – and then planning for each of those large-scale outcomes.
Everything Is Predictable is an engaging and interesting history and exposition of how Bayesianism is critical in certain specific scientific, medical, and other applications. Bayes helps to show us non-intuitive results in areas such as vaccine efficacy, cancer screening, and the like.
But it’s pointless even for issues such as one Chivers himself brings up at the beginning of the book: the probability of Russia invading Ukraine. He thinks the problem with it is “[W]hat’s your base rate? The average number of land wars in Europe per year? The average number of Russian invasions of Ukraine per year? It’s a subtle art—picking an appropriate reference class to compare your example with.”
But that’s not the real issue. The real question is, why predict this numerically at all? If there are two basic outcomes, or three or four, you already know enough to decide it’s something of huge importance. And you can’t get anywhere near 0% or 100% probability, so you’re going to have to prepare for both eventualities, no matter what.
So for strategic decision-making, this Bayesian approach is almost completely useless. And in that respect, it is a worthy addition to the Canon of the Cult of Prediction, which is extensively documented in my forthcoming book, Fatal Certainty: How a Cult of Prediction Made the 21st Century an Era of Strategic Shock, and How Rigorous Imagination Could Bring Us Back.
