Despite its flaws, Noise, the blockbuster from Daniel Kahneman, Oliver Sibony, and Cass Sunstein, is an important book that both informs and entertains.
The authors are at their best when describing the shocking noisiness of human decision making, not just among different observers in the same circumstances, but even by single observers in the same circumstances. Different judges, for example, hand down radically different sentences, on average, for the same cases; most amazing of all, sentences imposed by the same judge in nearly identical cases will vary from occasion to occasion and even by time of day.
Pro tip: avoid being sentenced more than two hours after the judge’s last meal.
The author engagingly describe the same inter-and intra-observer variance in multiple domains: hiring decisions; fingerprint interpretation; diagnoses by a wide range of medical specialists, most disturbingly in psychiatry. (Which won’t surprise any health care professional; after beginning his career as a ground-breaking child neurologist, Sigmund Freud spent the rest of his life in the land of untestable hypotheses.)
The authors intersperse these examples with a theoretical framework that breaks down noise into three domains. In the setting of judges’ sentences, for example, they are:
- level noise: the variance of the average sentences handed down by different judges across all cases, i.e., “bleeding heart” versus “hanging” judges;
- stable pattern noise: sentencing variance among cases according to the type of crime. I.e., some judges are hardest on drug cases, others on white collar crime; and
- occasion noise: sentencing variance at different times by the same judge the same type of case.
Confusingly, they occasionally refer to “system noise” as the total of all three types, and “pattern noise” as the sum of the last two types.
In addition to the noise as subsumed by the above three terms, they also describe bias, the systemic error of observers’ forecasts. Stock analysts, to use an example familiar to this publication’s audience, often produce widely varying estimates of a given company’s future earnings, but the average of their estimates for a particular company is usually overly optimistic (particularly those of sell-side analysts), so even if their system noise – the sum of the above three terms – was eliminated, their forecasts would still contain a significant overestimation error.
Worst of all, although these definitions are scattered throughout the book in varying and repetitious fashion, the authors never arrive at a simple equation that defines the total forecasting error – mathematically, the mean of squared errors (MSE).
It appears to be as follows:
MSE = [bias]2 + [level noise]2 + [stable pattern noise]2 + [occasion noise]2
(In fairness, the authors do represent this equation in a complicated graphic on page 211, but the reader has to turn the page sideways, squint, and then derive it from its complex depiction of nested Pythagorean triangles; not until the book’s midpoint is it revealed that in most cases, stable pattern noise is the MSE’s dominant component.)
The book’s central conundrum is embedded in its title, and in the beginning of one of the later chapters: “If the goal is to reduce noise or decide how and whether to do so . . .” Is the primary goal of the decision maker or forecaster really the avoidance of noise, or is it the avoidance of error? For the former is merely a symptom of the latter. Imagine, for example, if three distinguished infectious disease specialists produced a textbook on their subject entitled “Fever.” Fever, of course, is a merely a nonspecific symptom of an extraordinarily wide variety of illnesses. Similarly, noise is the non-specific marker for a dazzlingly wide variety of forecasting errors, the elucidation of many of which won Professor Kahneman his economics Nobel, and which he described in far more compelling fashion in Thinking, Fast and Slow. One can treat fever with aspirin or acetaminophen, but the competent physician directs her efforts at identifying and treating the underlying cause of the fever, not the pyrexia itself.
The authors do a good job of identifying these underlying noise-producing behaviors, but, again, here is where the focus on noise, as opposed to inaccuracy, causes problems. Overconfidence, surely suspect A in most forecasting crimes, gets only the briefest of mentions, and they don’t bother at all with another major source of error, the human preference for narrative over data and fact. They do better with groupthink, which they describe as an “informational cascade,” but inexplicably omit Asch’s famous experiments on conformity, surely the most intuitive way to understand this phenomenon, as well as the recent work on its evolutionary roots and how the fascinating results of Asch’s experimental paradigm vary across cultures.
If your time is limited, the book is repetitive enough that you can begin at Chapter 21, at its 60% marker, which introduces Tetlock’s forecasting recommendations, with later chapters describing “decision hygiene”: considering the “outside view,” that is, the base rate of outcomes; the aggregation of opinions; and ensuring the independent judgments of participants.
Uniquely, the book’s most intriguing sections are its three Appendixes, which describe in detail how large organizations can deploy “noise audits” for sources of the three systemic noise components listed earlier. Skepticism of this procedure, however, is in order; it’s impossible to forecast whether noise audits will revolutionize organizational decision making or, in the fullness of time, wind up as yet one more example of new age management woo woo.
Despite its flaws, you should read this book, if for no other reason than that one cannot learn enough about humankind’s dismal ability to forecast and decide. And after you’re done, dive into Tetlock’s magisterial Expert Political Judgment, surely the best diagnostic exposition of the range and sources of forecasting error ever written, and its therapeutic follow-on, Superforecasting.
William J. Bernstein is a neurologist, co-founder of Efficient Frontier Advisors, an investment management firm, and has written several titles on finance and economic history. He has contributed to the peer-reviewed finance literature and has written for several national publications, including Money Magazine and The Wall Street Journal. He has produced several finance titles, and four volumes of history, The Birth of Plenty, A Splendid Exchange, Masters of the Word, and The Delusions of Crowds about, respectively, the economic growth inflection of the early 19th century, the history of world trade, the effects of access to technology on human relations and politics, and financial and religious mass manias. He was also the 2017 winner of the James R. Vertin Award from CFA Institute.
Read more articles by William Bernstein
Despite its flaws, Noise, the blockbuster from Daniel Kahneman, Oliver Sibony, and Cass Sunstein, is an important book that both informs and entertains.
