Work

Weapons of math destruction

Year: 2018
Type: book-review
Abstract: Algorithm and data models are penetrating almost every area of our lives.Search engines tailor their ads based on information established by our online history.Grocery stores track our purchases and develop algorithms to tempt us with coupons.Data models determine whether we are hired, fired, and/or promoted.On the surface, these models may appear to be efficient and fair, but as Cathy O'Neil forcibly argues in Weapons of Math Destruction, they can be far from value-neutral math equations.Rather, they are embedded with hundreds of assumptions, are often flawed, and can draw faulty conclusions with devastating effects for society.To make matters worse, weapons of math destruction (WMDs) do not harm all individuals equally: poor, marginalized, and vulnerable communities are at a higher risk of suffering negative consequences from WMDs because data model designers do not consider their interests and because they have less power to fight against negative outcomes.O'Neil is a big data analyst who is exposing the messy underbelly of the field.Before writing this book, O'Neil earned a Ph.D. in Mathematics and spent years building models for finance companies and internet advertising firms.While creating these models, she saw many of the shortcomings of the analytical tools and how the techniques used to build the models can be incomplete, sloppy, and discriminatory.At the beginning of Weapons of Math Destruction, O'Neil confesses that she is not an evangelist of big data models and throughout the book we learn that she is a proponent of severely curtailing the implementation of WMDs.In fact, O'Neil states that 'in the name of fairness, some of the data should remain uncrunched' (150).It is rare for a data scientist to propose leaving data un-crunched and this book is a striking counterbalance to the wave of optimism over big data models.O'Neil's critique of WMDs is based on three broad claims: data scientists tend to use bad proxy variables; the models can produce negative feedback loops; and data scientists often draw conclusions from unsound statistical techniques.First, in order to create reliable models, data scientists need to accurately measure the phenomena they desire to produce or otherwise affect.However, even in today's data rich society, direct measures of the desired phenomena are lacking, and therefore WMDs frequently rely on proxy data to make estimates.In a perfect world, the proxy data would be a suitable substitute for the ideal measurement, but often the proxy variable is inaccurate and yields a questionable assessment.For example, many algorithms classify individuals as either risky or safe.Instead of actually assessing each individual person, however, these models look at general characteristics, such as where the person lives, their social network, and their credit scores.These proxy variables do not actually measure whether a person is risky; rather, the models correlate proxy factors with risk.Thus, depending on the sensitivity of a given model, it can falsely categorize large portions of the population as risky, when in fact they are very safe.Compounding the problem of using proxy variables, O'Neil's second issue is that analysts rarely update the models to make them more accurate.By failing to iteratively build WMDs based on trial, error, and correction, she argues that data analyst can create strong, negative feedback loops.Third, O'Neil claims that the creators of the models use statistically This article has been republished with minor changes.These changes do not impact the academic content of the article.
Language: English
Open Access status: bronze
OpenAlex ID: W2887489621
Cites:
Cited by: 516
Related to: 8