Is the market efficient? Of course not – not exactly, or not even close, depending on your point of view. However, the efficient market hypothesis (EMH), while self-evidently not quite correct, has remained surprisingly resistant to overturning. The reason is that, as MIT professor Andrew W. Lo says repeatedly in his new book, Adaptive Markets, “it takes a theory to beat a theory.” And, up to this point, there has been no alternative theory that can substitute for the EMH if the latter is found wanting.
In the modern study of capital markets there have been two kinds of innovation: (1) insights that contribute to our ability to make better judgments, as found in the brilliant work of Fischer Black, Peter Bernstein, Marty Leibowitz, and many others; and (2) true hypotheses and theories, which are testable and, as I’ll explain shortly, falsifiable. The latter are few and far between. Andrew Lo’s book aims to present a new theory of capital markets, but, while it does not really do so, it is full of brilliant insights into behavior, evolution and the ways in which these factors help us to better understand how capital markets work.
What is a hypothesis?
Karl Popper, the great 20th century philosopher of science, said that a hypothesis is a statement, intended to explain a set of observations, which can be falsified. It only takes one exception to a supposedly universal rule to prove it wrong: the existence of one black swan, Popper said, proves that the rule “all swans are white” is incorrect.[1]
By this standard, Lo’s adaptive markets hypothesis (AMH), despite its name, is not a theory or hypothesis. It is a set of observations about human nature and, by extension, the behavior of markets. Lo delivers a detailed and thoughtful critique of the EMH in particular and modern finance in general, but no new theory and no revolution in financial thinking. Along the way, however, he introduces a great deal of challenging and informative material, and for that reason the book is worth reading.
The EMH in context
The EMH, in contrast, is a proper hypothesis or theory. It is a theory of price. It says that the price of a security reflects all available information. (If this is true, you cannot beat the market on a risk-adjusted basis except through pure luck, but that is a consequence of the EMH, not an essential element of it.) If all available information can be reduced to a set of period-by-period – say, annual – cash-flow forecasts, then the price of the security is the discounted present value of those cash flows, where the discount rate reflects the amount of risk inherent in the forecasts.
The EMH says that the market performs this task correctly. It does not say that every individual does it properly, only that in aggregate the answer is correct: “the price is right.”
The EMH can be falsified by finding even one example of an abnormal profit opportunity in the market that is not arbitraged away. There are so many such examples that no one considers the EMH to be literally true any more.
Spoiler alert: Here’s the adaptive markets hypothesis
The only thing that is lacking, then, is an alternative to the EMH. It is not good enough to say “the market is not efficient.” How, then, are security prices formed if not by market participants collecting all available information and agreeing on a price so they can trade? Do they ignore certain types of information? Do they disagree on the importance of each piece? Of course they do, but what is the mechanism?
Here is Lo’s presentation of the AMH:
The basic idea can be summarized in just five key principles:
- We are neither always rational nor irrational, but we are biological entities whose features and behaviors are shaped by the forces of evolution.
- We display behavioral biases and make apparently suboptimal decisions, but we can learn from past experience and revise our heuristics in response to negative feedback.
- We have the capacity for abstract thinking… predictions…based on past experience; and preparation for changes in our environment. This is evolution at the speed of thought, which is different from but related to biological evolution.
- Financial market dynamics are driven by our interactions as we behave, learn, and adapt to each other, and to the social, cultural, political, economic, and natural environments in which we live.
- Survival is the ultimate force driving competition, innovation, and adaptation.
These are generalizations that describe life in the real world, but they do not constitute a theory. A scientific theory or hypothesis (there’s a difference but I’m ignoring it for the moment) can usually be reduced to a statement something like, “If we perform experiment A, we expect outcome B to be the result, because of thus-and-such.” The “because,” the mechanism, is important; it cannot be a mystery process.
In the sciences, if we drop a ball from a tower near the surface of the Earth we expect it to accelerate by an amount of 32 feet per second each second. In the social sciences, if we subsidize a good we expect to get more of it, and if we tax a good we expect the economy will produce less of it. The predictions are less precise in the social sciences – after all, it’s people whose behavior we’re trying to predict – but there must be some predictive ability for the theory to have any use at all.
Classical finance and behavioral or evolutionary finance
From Markowitz’s 1952 discovery of portfolio optimization to 1979, when Daniel Kahneman and Amos Tversky published their seminal work, Prospect Theory: An Analysis of Decision Under Risk, we were on a path that used neoclassical economics, including its assumption of rationality, along with the new and powerful tool set of data and computers, to understand capital markets.[2] We made great progress, and we had insights worthy of Nobel Prizes – those of Harry Markowitz, William Sharpe and Eugene Fama.
But, in 1979, Kahneman and Tversky opened a parallel door by asserting the obvious: That we are not balls dropping from a tower, but organisms. As such, we are fallible and prone to biases and misunderstandings. Their parallel door leads to a different set of insights and that is what this book is about – it’s a compendium of what we’ve learned about finance in the 38 years since Kahneman and Tversky published their landmark work, augmented by Lo’s unique take on the way that evolution and survival fit into the behavioral story.
Evolution and adaptation
How, then, does the AMH translate information in the economy to a security price? After all, that is the question we seek to answer when forming and testing hypotheses about the way financial markets work. Lo replies,
“Prices reflect as much information as dictated by the combination of environmental conditions and the number and nature of ‘species’ in the economy.” (Lo [2005], p. 36)[3]
What he calls “species” are the various market participants in the financial zoo: issuers, market makers, security analysts, long-term investors and so forth. Each struggles, not to maximize profits, but to survive, as real species do in an ecosystem.
The bottom line – the prediction that AMH makes – is that the prices of some securities are more efficient than others. Nearly everyone can agree on what the price of a 10-year Treasury bond should be; all the species use the same bond math. Fewer people will agree on what the price of an ancient Greek vase should be; the “issuer” has been dead for thousands of years, there are only a few potential buyers, and “market makers” – dealers, museums and so forth – have their own agendas. So there should be more alpha or profit opportunities in assets that resemble the vase than in those that resemble the Treasury bond.
This assertion is not directly testable in the way that the EMH is, but it is certainly more realistic.
The arithmetic of active management
The bad news (this is my interpretation, not Lo’s): Because William Sharpe’s “Arithmetic of Active Management” [1991] applies always and everywhere,[4] if there are more profit opportunities in less efficiently priced asset classes then there are also more loss opportunities. No matter how inefficiently priced an asset class, simply being an active manager does not guarantee you a profit. You have to extract the profit from other active managers who are trying just as hard, are likely to be just as smart, and need the money just as badly as you do.[5]
Evolution and the origin of behavior
Now that I’ve gotten that off my chest – there’s no theory! – I feel obliged to emphasize that there’s a whole lot of fascinating material in Adaptive Markets. Let’s focus on one topic, the evolutionary origin of human behavior. (Lo has no small ambitions.)
“Probability matching” is betting on an event in proportion to the likelihood of it occurring, instead of betting on the most likely event every time. For example, if an urn contains three-quarters red balls and one-quarter black balls, most people will bet on the ball being red three-quarters of the time instead of all the time (the latter strategy being the one most likely to win).
Although probability matching seems irrational, people do it all the time. So, writes Lo, do “ants, fish, pigeons, rats, and [non-human] primates.” What’s wrong with them?
The trouble with tribbles
Lo, with a co-author (Tom Brennan), wrote an ingenious paper in 2009, summarized in Adaptive Markets, which answered this question.[6] Brennan and Lo set up a simulation in which tribbles, mythical furry creatures from Star Trek, live or die according to geography and the weather. Each tribble has three offspring.[7]
Tribbles can choose to live in a valley or on a mountain. If they nest in a valley and the weather is sunny, all their offspring survive; if it rains, they drown. Conversely, if they live on a mountain and it rains, their offspring survive; if it is sunny, they die of sunburn (I guess). It is sunny 75% of the time, it rains the other 25% of the time, and tribbles do not know how to predict the weather.
Now, how should a tribble decide where to live? If it simply wants to maximize the number of surviving offspring, it lives in the valley – an undiversified bet that is right 75% of the time. There is no strategy that is right more often.
But if the community of tribbles wants to maximize the number of its surviving offspring, the best strategy is a surprising one: it diversifies its bets, with some tribbles settling in the valley and others on the mountain. Exhibit 1 shows the number of surviving tribbles, generation by generation, for various diversification strategies (values of ƒ, shown across the top of the exhibit, ranging from ƒ = 0.20, where 20% of the tribbles settle in the valley, through ƒ = 1, where all of them do.
Exhibit 1
Number of tribbles in each generation for tribble populations with various diversification strategies (ƒ-values)
Note: f is the proportion of a given tribble population that locates in a valley.
Source: Lo [2013], p. 274.
At first, valley tribbles multiply faster than any of the other populations, because it rarely rains. This success reflects the optimality of their decision, made in a single-period framework by individuals trying to maximize their own reproductive success. But, in the 14th generation, it rains and the entire valley tribble population is wiped out.
After 25 generations, only the communities with diversified strategies have survived at all. The tribble community with the most survivors is the one that allocates 75% of its population to the valley and 25% to the mountain – matching the probability of sunshine, also 75%.
Probability matching! Remarkably, through this little thought experiment, Brennan and Lo derive not only probability matching, but also portfolio diversification and kin altruism – all complex behaviors widely observed throughout the animal and plant kingdoms – from the most basic principles of natural selection, or evolution. (Kin altruism is the tendency of individuals to sacrifice their personal interest to help others who are genetically related, with the degree of sacrifice typically proportionate to the degree of relatedness.)
Bounded rationality and “satisficing”
In another tour de force, mercifully shorter and easier to explain, Lo – this time without a co-author – studied “The Origin of Bounded Rationality and Intelligence.”[8] Bounded rationality is the human tendency to limit the amount of effort spent in trying to behave rationally. While Lo’s bounded-rationality paper is quite extensive, an example from Adaptive Markets provides the gist of it. Each morning, Lo chooses an outfit to wear to work. With five jackets, ten pairs of pants (five of which match the jackets), 10 shirts, 20 ties, and a number of accessory items, Lo has 2,016,000 possible outfits to choose from each morning.
Does he optimize by estimating the utility of each of the two million outfits, perhaps decrementing the utility of a potential outfit if he’s recently worn it, and choose the single best one? No – if he did that, he’d never get dressed. Instead, he “satisfices,” the Nobel Prize-winning economist Herbert Simon’s term for figuring out what solutions are good enough, as opposed to formally optimal.[9] “I use a variety of heuristics,” Lo writes, “to balance the cost of evaluating different combinations against the desire to get to work on time.” One of Lo’s heuristics is to avoid wearing a yellow striped tie with a red striped shirt: a colleague thought he looked ridiculous.
How do you know when your solution is good enough?
You don’t. You develop rules of thumb by trial and error. You usually don’t know whether a decision is truly optimal. Over time, though, you experience positive and negative feedback from those decisions, and you alter your decisions in response to this feedback. In other words, you learn and adapt to the current environment.
That’s what investors do too, and it’s one reason why the market is not perfectly efficient – but not grossly inefficient either (with easy profit opportunities everywhere).
An intellectual adventure
Puzzles like this, along with highly creative and sophisticated solutions to them, make up much of Adaptive Markets. It’s an intellectual adventure – a very long one – for people whose interest in markets and behavior has led them to pursue a liberal arts self-education strategy. Readers who are undaunted by Richard Bookstaber’s Demon of our Own Design and Nassim Taleb’s Fooled by Randomness will love it. More practical minds may have some difficulty with it.
Andrew Lo is well prepared to lead such an adventure. He is sufficiently well read that his work on just one topic – the evolutionary origins of behavior – brings together five quite separate threads of scientific literature, all but one outside finance:
- Behavioral economics and finance;
- The psychology and cognitive science of Amos Tversky and Daniel Kahneman;
- Evolutionary psychology and sociobiology;
- The evolutionary game theory of John Maynard Smith; and
- Behavioral ecology, which derives from the work of Charles Darwin, R. A. Fisher, J.B.S. Haldane, and other intellectual giants of the last century and the one before.
Lo’s conclusions are also suitably lofty, given this list of influences: “We derive risk aversion, loss aversion, probability matching, and randomization from evolution.”[10]
So that’s where human behavior comes from. We might have guessed that, but it’s nice to have a proof, or what passes for proof in the social sciences.
Practical applications
Does Lo provide anything practical? I’ve emphasized some of Lo’s more exotic ideas, partly to show how good his work is, but also because his flights of fancy regarding tribbles and tie-wearing are more fun than his financial advice. But the intellectual virtuosity with which he addresses practical concerns is just as impressive.
He discusses financial fraud, ways to “fix” finance, hedge funds (“the Galapagos Islands of finance,” where speciation takes place at warp speed) and events such as the bizarre “quant crash” of August 2007. (The quant crash was, it turns out, a temporary failure of statistical arbitrage, an under-the-radar momentum strategy that had grown huge as large banks and hedge funds adopted it.)
Adaptive Markets is encyclopedic, and any attempt to summarize it in a brief review will fall short.
Conclusion
At half the length, and with the claim of having created a new theory left out, Adaptive Markets had the potential to be a classic of the literature of science. Richard Dawkins’ The Selfish Gene is the archetype of this genre. In finance, Peter Bernstein’s Against the Gods and Nassim Taleb’s Fooled by Randomness are fine examples.
Lo is not quite as vivid a writer as these exemplars but he has a keen sense of what might interest thoughtful readers. The ideas in this capstone volume, summarizing an extraordinary career, may be better conveyed in the journal articles, speeches and classes from which the book has been constructed; at least, in those they are separated into manageable pieces.
Fans of Lo’s thinking should hear his speeches – his delivery is truly exceptional.[11]
By reading Adaptive Markets, investors and their advisors will benefit from a better understanding of the reasons, based in biology and human nature, that the market is not efficient. In fact, the market is so full of gross inefficiencies that active management should be like shooting fish in a barrel. But it’s not, because of competition and arbitrage. However, competition and arbitrage do not eliminate all profit opportunities. Keep analyzing securities, managers, asset classes, or whatever enables you to take advantage of your expertise.
Laurence B. Siegel is the Gary P. Brinson Director of Research at the CFA Institute Research Foundation and an independent consultant. He may be reached at [email protected]. Steve Sexauer provided extensive comments.
[1] Popper, Karl. The Logic of Scientific Discovery (1959). Popper’s statement, above, is the origin of the concept of the “black swan” that Nassim Nicholas Taleb used to great effect – but Taleb uses it to mean a highly improbable event with a large impact, while Popper uses it to mean a single observation that invalidates or falsifies a generalization (quite a different meaning). Taleb, Nassim N. 2007. The Black Swan: The Impact of the Highly Improbable, New York: Random House.
[2] Kahneman, Daniel, and Amos Tversky. 1979. “Prospect Theory: An Analysis of Decision Making under Risk.” Econometrica, Vol. 47, no. 2 (March), pp. 263-291. Versions of this article were made available to the public earlier, but it is customary to date an intellectual innovation from the time it is published in a refereed journal.
[5] Some observers try to evade the arithmetic of active management by saying that, for less homogeneous and less liquid asset classes, there’s no benchmark. While that may be technically true, there’s still an opportunity cost of capital that you have to exceed to make an economic profit – that is, there are alternative uses for the money, and that’s the “benchmark.”
[6] Brennan, Thomas J., and Andrew W. Lo. 2011. “The Origin of Behavior.” Quarterly Journal of Finance, Vol. 1: 55–108.
[7] The number of offspring is important because individual tribbles can decide to “move house,” so more offspring means the potential for greater diversification (in this case, in exposure to both mountain and valley conditions). Tribbles reproduce asexually, removing the need to model their mating behavior. Making up stories like this is what economists do…really. The Star Trek episode in which tribbles were introduced was also called “The Trouble with Tribbles.” It aired on December 29, 1967 and was written by David Gerrold.
[9] Simon, Herbert. “A Behavioral Theory of Rational Choice.” 1955. Quarterly Journal of Economics, vol. 69, no. 1: pp. 99-118.
Read more articles by Laurence B. Siegel
