No More Stupid Forecasts!

Watching experts predict the future is like watching professional wrestling. You assume everybody knows it’s a put-up job but can’t resist it anyway. Then you discover that most people don’t even know it’s a put-up job in the first place.

Dan Gardner’s book Future Babble takes the sport of expert prediction apart piece by piece, showing why it’s phony, why people still pay close attention to it and why people (including the experts themselves) continue to believe in it. Along the way, the author — who is a real “fox,” as I’ll explain later — fills his pages with enough interesting information and anecdotes to keep us reading with pleasure.

The basic data

Gardner’s book is anchored in the academic work of Philip Tetlock, a professor of management and psychology at Wharton. Beginning in 1984 and continuing for at least 20 years, Tetlock performed an exhaustive long-term controlled study of expert predictions. He collected 27,450 well-defined expert predictions, along with the experts’ subjective estimates of their probabilities of occurring. According to Gardner, Tetlock’s experts were very diverse – from different fields, political leanings, institutional affiliations and backgrounds – and they were asked “clear questions whose answers can later be shown to be indisputably true or false.”

Then Tetlock assessed whether the experts’ predictions performed any better than a dart-throwing chimpanzee. The result — perhaps not surprisingly — was that for the most part, they did not. But some experts’ predictions were better than others.

The measurement system

Tetlock used an elaborate measurement system to determine how good the predictions were. Here is a simple example:

Everybody knows — or assumes — that the chance of a tossed coin coming up heads or tails is 50%.

It seems random, but the result of a coin toss is completely determined by forces that obey the laws of physics: the strength and placement of the initial impulse that sent the coin skyward; the air currents in the space through which the coin travels; the coin’s contours and distribution of mass; the shape, hardness and frictional qualities of the surface on which it lands; and the interactions of all of these factors.