Newsweek recently published its list of the best 2,000 high schools in America. Looking over the list of schools in my home state of Massachusetts, I was very surprised that my children’s school, Natick High, was ranked significantly higher that my alma mater, Newton North High School. By all standardized scoring measures – SATs, ACTs and AP scores – Newton North beats Natick hands down. So why was Natick ranked higher?
It turns out that 25% of Newsweek’s ranking is based on the percentage of graduates going on to college. Natick’s 96% college-bound rate was much higher than Newton North’s 82% figure. However, it is wrong to conclude that Natick produces more college-bound students. Newton North’s numbers include students in its in-house vocational school, many of whom will go on trade schools, apprenticeships or directly into the workforce. Trade-minded Natick students, on the other hand, go to a separate technical-vocational high school.
Welcome to the world of numerical obfuscation, a topic covered in an informative and surprisingly entertaining way in Naked Statistics: Stripping the Dread from the Data, by Charles Wheelan, a senior lecturer and policy fellow at the Rockefeller Center at Dartmouth College.
In a recent interview with Wheelan, a former correspondent for The Economist and author of several other books , we discussed Naked Statistics and the lessons it offers to financial advisors, whose decision-making processes are influenced by the flood of economic and market data posted every day.
“You can’t be an investment advisor without looking at data, and you can’t look at data without using statistics, because statistics really are nothing more than the tools for interpreting and simplifying huge amounts of information,” Wheelan said. “But the very process of simplifying also introduces some potential error, depending on the measure you use.”
The book offers user-friendly overviews of common statistical and probability concepts advisors encounter on a daily basis as they trudge through performance figures, economic commentaries and analyst reports. A number of chapters offer how-to guides for those who, for example, would like to know how to manually calculate standard deviations and correlation coefficients.
But Naked Statistics is much more than a “stats for dummies” primer, due to Wheelan’s cautionary tales that illustrate how selective analysis, manipulation of data and sloppy research methodologies can lead to biased and misdirected conclusions posing as numerical truths.
For example, the deceptively simple decision of whether to use mean or median as a measure for analyzing personal income growth over the past decade can lead to dramatically different interpretations, even when the same core data are used. Mean data show that, except for a drop during the Great Recession, personal income grew during this period. If you use median income as the measurement, you see that income growth has largely remained flat. The reason for this discrepancy is that the mean income results are skewed upward by a disproportionate rise in income among the very wealthy.
But even this measure doesn’t tell the full story of the growing gap of income growth between the rich and poor in America. For that, you need to divide the income data into percentile ranges and plot them separately. Using weekly wage data from 1979 to 2009, Wheelan provides an example of this approach that shows that those in the top 25% wage percentiles grew wealthier during this period, while the vast majority of American workers experienced little or no wage growth.
Yet, even when the results of statistical analysis are reliable, the truth they represent might be contradicted by other data. In the book, Wheelan uses the age-old question, “How healthy is the manufacturing sector?” as an example. He provides a chart that shows manufacturing output and manufacturing employment levels over the past decade. If you only look at output, it appears that U.S. companies are producing and exporting more products than ever before, except for a sharp dip during the Great Recession. However, you also see a steady decline in manufacturing jobs during the same period, suggesting an industry in decline.
How do you reconcile those two stories, both of which are correct? By using a third statistic – productivity, which has grown enormously in the past decade as manufacturers make more products with fewer workers. “Sometimes you need several different data points to tell a more nuanced story about what is happening in the economy or in specific industry sectors, which is critically important for advisors who are evaluating the risk/return potential of these sectors,” Wheelan said.
Wheelan also examined the dangers of stochastic forecasting models (like Monte Carlo simulations) and their role in exacerbating the market meltdown of 2008-2009. He contends that these methodologies underestimated value-at-risk (VaR) thresholds because their assumptions were based solely on historical market conditions, rather than anticipating the effects of a potential economic catastrophe. For example, many highly sophisticated VaR models failed to include the probability of the effects of a nationwide collapse in the housing market, one of the key causes of the Great Recession.
“These flawed VaR models encouraged investment bankers and traders to make excessively risky decisions, many of them involving junk mortgages and derivatives,” Wheelan said. “By failing to safeguard against worst-case scenarios, the industry hastened its fulfillment.”
“The financial industry’s negligence during this crisis is the equivalent of San Francisco city planners establishing earthquake-resistant construction standards solely on the frequency and level of damage caused by past earthquakes over the past century, rather than anticipating what could happen if and when the big one hits.”
Another subject that will be of interest to advisors is Wheelan’s scrutiny of the increasing use of rating systems that distill wide arrays of data points into often meaningless numbers or rankings. As demonstrated in the Newsweek high school rankings, some of these rating systems use flawed methodologies.
For financial advisors who routinely rely on Morningstar ratings, Lipper rankings and assorted ‘best funds’ lists, it’s important to critically assess if their methods align with advisors’ own due-diligence standards. For example, one vendor may give higher weightings to a fund’s five-year performance record, while another may place greater emphasis on fee efficiency.
Wheelan also recommended that advisors cast a skeptical eye toward fund companies’ promotion of their top funds.
“Many fund families start a large number of funds and eventually shut down those that underperform, leaving only a few benchmark-beaters they heavily promote as the firm’s spotlight funds,” he said.
Wheelan likened this to an exercise he does with this students.
“I’ll have everybody flip a coin. Those who flip heads flip it again, while those who flip tails sit down,” he said. “After several rounds we’ll end up with one person standing who has flipped five heads in a row. So I ask him or her, ‘How did you become such a good flipper of heads?’ In the investment world, a fund’s company’s top-performing funds may simply be those that happened to ‘flip five heads in a row.’”
Wheelan also takes on the academic and medical-research industries, He questions the honesty of those who only publish results when studies show a positive correlation between two factors (such as the consumption of red wine and lower levels of heart disease), while previous studies that failed to show a correlation go unpublished. Such studies may generate social media buzz and spur sales of Bordeaux and Merlot, but someone planning to up their red wine consumption should view these headlines with skepticism.
Even if the sponsor of the research isn’t the Pinot Noir Growers Association, the methodology itself might be flawed. For example, the individuals participating in the test might not be representative of the population as a whole: If the majority of wine-drinking test subjects are vegans or marathon runners, they’re already likely to have a lower incidence of heart disease.
While Naked Statistics won’t magically turn you into a data-crunching numbers geek, its undressing of statistical complexities and its useful strategies for evaluating their reliability make it a must-read for those seeking greater truth behind numbers.
Jeff Briskin is Director of Marketing for Advisor Perspectives and president of Briskin Consulting, a Boston-area marketing and creative services firm. He can be reached at [email protected].
Read more articles by Jeff Briskin
