Depending on whom you ask, inequality is driven by globalization, tax policies, crony capitalism or some other macro-economic force. But what if something more sinister is preventing poor people from advancing?
“[I]f a prosecutor attempted to tar a defendant by mentioning his brother’s criminal record or the high crime rate in his neighborhood, a decent defense attorney would roar, ‘Objection, Your Honor!’ And a serious judge would sustain it. This is the basis of our legal system. We are judged by what we do, not who we are.”
So asserts Cathy O’Neil, author of the recent book, Weapons of Mass Destruction: How Big Data Increases Inequality and Threatens Democracy, available from the link on this page. But when a recidivism algorithm evaluates whether or not a prisoner is a risk to society and should be paroled, the algorithm does precisely what is not allowed in court. It evaluates the subject based on his background, his family, and the neighborhood he grew up in. And it does this in an opaque manner, using computer codes that are “proprietary” and inaccessible to the prisoner or anyone else because they belong to for-profit companies.
Recidivism algorithms are just one example among many in O’Neil’s book of Weapons of Math Destruction, or WMD. Another example is auto insurance algorithms that determine how much a driver will be charged based on his credit rating rather than on his driving record. Why? Partly because credit ratings are so easily accessible, but also because low credit ratings correlate well with stress-induced behavior; people with low credit ratings are usually under stress. Whether that means that low credit ratings actually correlate better with a driver’s future driving record than the driver’s past record – which seems unlikely – we don’t know, because the algorithm and the reasons for its construction are not transparent.
The chief message of O’Neil’s book is that these algorithms – which are becoming increasingly standardized and widely-used –tend to lock poorer people into poverty. For example, when a person has to pay a higher insurance rate because of a low credit rating, or is denied a job – because credit ratings are used in algorithms that rate potential employees too – that person’s poverty will be sustained, or they will become poorer, and hence their credit rating will fall further. Thus, poverty will be more difficult to climb out of.
And if a person fights back against an algorithm – as some teachers fought back when the algorithm that evaluated them produced obviously erroneous results – the evidence that the algorithm was wrong may be considered “soft,” even if it is perfectly clear to anyone that the evaluation was wrong. By contrast, the computer algorithm is considered “objective” and thus infallible by assumption. “The human victims of WMDs, we’ll see time and time again, are held to a far higher standard of evidence than the algorithms themselves,” O’Neil says – algorithms whose details are unavailable to anyone, least of all to their victims.
O’Neil admits that when potential employees, or applicants for loans, or candidates for parole, were evaluated in the past, they may have been evaluated by the subjective judgments of biased individuals. But at least those judgments, she says, were diverse across different employers, parole officers, etc. This might allow an individual to crack open a door to squeak through the bias against them, either by finding an unbiased human evaluator, or by finding one who happens to be biased in the individual’s favor. But when the bias is set in concrete in standardized, opaque algorithms and used universally for its supposed “objective” merit, a person who – for example – has a low credit rating will be unable to find a way around it.
Extrovert among introverts
For a person with a PhD in pure mathematics, O’Neil is, from my perspective, highly unusual. Mathematicians usually appear in old jokes about bashfulness and introversion, jokes that are often about the relative “sociability” of engineers – for example, “a mathematician looks at his shoes when he talks to you, but an engineer looks at your shoes.” To do mathematics well requires an extreme degree of introspection, wrestling with a problem inside your mind sometimes for months or years.
But O’Neil is an engaging extrovert. After giving up her academic career in mathematics to take a job at the elite hedge fund D. E. Shaw, then leaving it, as well as her next job at the risk-management consulting firm RiskMetrics, in disenchantment, she launched a widely read blog at mathbabe.org. At the same time she launched a sister column titled “Ask Aunt Pythia” – a spoofy advice column in which she solicited questions about social problems, especially about sex. And the answers were both excellent and fun.
At about the time she launched her blog she became a data scientist, working for a “big data” company. She also paid a visit to Occupy Wall Street. She believed their hearts were in the right place, but she found that they knew nothing about finance, so she joined their movement in order to instruct and help them.
It was in her data scientist role that she discovered the widespread dangers of opaque algorithms and began writing a book about it.
O’Neil’s experience in finance
When I interviewed her by Skype, however, I was particularly interested in O’Neil’s experiences in the financial industry at D. E. Shaw and RiskMetrics. I was appalled to realize that while her experiences in that industry were virtually identical to mine in many ways, mine had occurred many years earlier; nothing seems to have improved over all these years.
She joined D. E. Shaw in June 2007, just as things were starting to look a little shaky in the collateralized debt obligation (CDO) market. The group of people she worked with was not like the group in “The Wolf of Wall Street,” the movie that, although it didn’t say so, was really more about the penny stock market scams of the 1980s than about the 2000s – or those in Sam Polk’s book “For the Love of Money” (reviewed here).
No, her colleagues were mostly straight-laced young Eastern European men with families, who didn’t drink much or do drugs. But they also weren’t particularly reflective about what good their activities were doing for society – or for anyone other than themselves and their own firm for that matter. O’Neil, with a different perspective – which she attributed to being a mother – became disenchanted by contemplating whose wealth her firm’s winnings were coming from. If the firm did indeed win at all, much of the winnings came from the pensions of many little people, the losers who were on the other side of the trades D. E. Shaw executed. So what good was this doing for society?
The 2008 crisis devastated D. E. Shaw as it devastated so many others, because of the total chaos in the markets. O’Neil decided she wanted to understand how financial risk is controlled, and moved to a job at RiskMetrics, the Morgan Stanley spinoff that invented the value at risk (VaR) model.
The VaR model is extremely simplistic. It assumes that every asset’s probability distribution of returns can be modeled as a normal distribution. O’Neil noted that this is completely unrealistic for credit default swaps (CDSs), whose distribution of results is very fat-tailed (i.e., the probability of extreme losses is much greater than it is in a normal distribution). So she proposed integrating a different and more realistic model of the distribution of returns for CDSs into the VaR model – and received almost no interest at RiskMetrics. Also, she found that RiskMetrics’ banking clients had little interest in whether the math was right – only in whether the exhibits looked good and fulfilled their regulatory requirements.
This rang true for me; good mathematics, realistically applied, is of little interest to firms with a reputation for “sophisticated” models. The fine details of whether their models are actually right or not, or even whether they are actually sophisticated or not seem to have little impact on their reputations for sophistication. So why bother?
O’Neil explained her frustration in this way, “I was forced to confront the ugly truth: people had deliberately wielded formulas to impress rather than clarify. It was the first time I had been directly confronted with this toxic concept.”
Welcome to the financial industry’s world of sales math. When the only criterion for success is whether the math looks sophisticated and helps the sale, whether it actually fits the practical problem and helps to solve it is of little consequence.
A zero-tolerance policy for Wall Street?
O’Neil points out that in the 1990s zero-tolerance campaigns in cities became popular in police departments, following the “broken windows” theory of criminologist George Kelling and public policy expert James Q. Wilson. The idea was that if low-level nuisance crimes like breaking windows and peddling small amounts of marijuana are allowed to continue in inner cities, they will remain breeding grounds for larger crimes. Therefore, according to the theory, a zero-tolerance policy would eliminate the breeding ground and put the brakes on crime.
This, however, has had the ultimate effect of filling prisons with young men largely of minority extraction from the urban areas that are most heavily patrolled, most of them for petty victimless crimes like possessing or selling small amounts of drugs.
Budget-strapped police departments have started to use predictive algorithms like PredPol and CompStat to predict where crimes are most likely to occur, so that they can deploy their limited forces there. This could be helpful, but minor crimes like vagrancy and selling small amounts of drugs are much more likely than major crimes like murder, assault and arson. And it is the poorer areas that have more crime, so deploying police to those areas results in more prosecutions for minor crimes.
“But,” says O’Neil, speaking from her Occupy Wall Street perspective, “how about crimes far removed from the boxes on the PredPol maps, the ones carried out by the rich? … Thanks largely to the industry’s wealth and powerful lobbies, finance is underpoliced. Just imagine if police enforced their zero-tolerance strategy in finance. They would arrest people for even the slightest infraction, whether it was chiseling investors on 401(k)s, providing misleading guidance, or committing petty frauds.”
As unlikely as this scenario seems, it may not be a bad idea to imagine it happening. Suppose that police were constantly trolling about banks and investment management and advisory firms, expecting trouble and usually finding it. This may provide a picture of what being a young man in an inner city area might be like.
What to do about it
Although O’Neil thinks – as I do – that math-based algorithms are used far too much in the social sciences, and far too much credibility and authority is assigned to them, she doesn’t think they’re all bad. On the contrary, she thinks if they are designed and implemented more effectively some of them could be beneficial. In fact, to help audit and improve these algorithms she has started her own company for that purpose.
Her example of a good use of algorithms is in baseball models like those described in Michael Lewis’s Moneyball. These models “are transparent and continuously updated, with both the assumptions and the conclusions clear for all to see. The models feed on statistics from the game in question, not from proxies [like credit rating]. And the people being modeled understand the process and share the model’s objective.”
The problem with so many of the commonly used citizen-oriented models is that the algorithms are opaque and inaccessible to the people being evaluated by them, and while they may have been created by means of statistical correlations, they aren’t routinely tested against new data and results to check that they are really any good. O’Neil explains, “Statistical systems require feedback – something to tell them when they’re off track.”
Among others, teacher evaluation models are particularly egregious. Numerous examples have arisen of the damage these algorithms do; yet they still remain largely impervious to criticism. In one case O’Neil describes, a teacher named Sarah Wysocki who had received plaudits from virtually everyone nevertheless received an astonishingly low score on an evaluation sold by a Princeton, NJ-based company with the impressive name of Mathematica Policy Research. When someone investigated the situation carefully, they discovered that the test score was based on “value-added” – how much the teacher improved the test scores of her students. When Wysocki’s students had entered her class, she was surprised to see how good their scores had been on their tests in the previous year. And yet when she became familiar with the students, she found they weren’t very good at all, and thus their test scores fell from the previous year. The investigator found strong circumstantial evidence that, responding in too-extreme a fashion to the prospect of such an evaluation, the previous year’s teacher may have countenanced cheating on the part of the students – and hence the high scores in the previous year. For that, this year’s teacher received a poor evaluation on the “value-added” score. This is not an atypical example of the dangers posed by these algorithms.
O’Neil’s bottom line is that, “This menace is rising. And the world of finance provides a cautionary tale.”
Michael Edesess is a mathematician and economist, a senior research fellow with the Centre for Systems Informatics Engineering at City University of Hong Kong, chief investment strategist of Compendium Finance and a research associate at EDHEC-Risk Institute. In 2007, he authored a book about the investment services industry titled The Big Investment Lie, published by Berrett-Koehler. His new book, The Three Simple Rules of Investing, co-authored with Kwok L. Tsui, Carol Fabbri and George Peacock, was published by Berrett-Koehler in June 2014.
Read more articles by Michael Edesess
