The Weapons of Math Destruction Edition

Summary

Slate Money Podcast – Weapons of Math Destruction Edition

Date: September 10, 2016
Host: Felix Salmon
Guests: Cathy O’Neil, Jordan Weissman
Main Topic: The impact and dangers of algorithms as detailed in Cathy O’Neil’s book Weapons of Math Destruction

---

Episode Overview

This special edition of Slate Money is dedicated to Cathy O’Neil’s then newly released book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. The hosts break down what makes an algorithm destructive, discuss real-world examples, and debate possible solutions to the dangers posed by opaque, unaccountable models. Throughout, O'Neil makes the case that too many algorithms—especially those that are secret and high-impact—are exacerbating inequality and reducing fairness, rather than advancing progress.

---

Key Discussion Points & Insights

1. Defining Algorithms and “Weapons of Math Destruction”

[04:20–08:00]

• Algorithm Basics:

  • Algorithms need two things: Data (training data to find patterns) and a definition of success (“objective function”).
  • Cathy O’Neil: “What the algorithm does is it works, looks for patterns in the history that you’re giving it in the data for when this definition of success actually occurs.” [04:35]

• What Makes a WMD?
  O’Neil outlines a three-part test:

  1. Widespread and High Impact: Used to make significant decisions affecting many lives.
  2. Secrecy & Unaccountability: The scoring process is opaque—people don’t know how or even whether they’re being scored, and there’s no appeals process.
  3. Destructive Feedback Loops: The model not only harms individuals but worsens the overarching problem it aims to fix.
  • Cathy O’Neil: “…[algorithms] are engendering these sort of negative feedback loops that actually undermine the original goals, which are often good goals, good intentions. But…they’re actually making the problems worse.” [07:52]

---

2. Example of a “Bad Algorithm”: Teacher Value-Added Models

[08:18–17:00]

• Flawed Models in Education:

  • Teacher “value-added” models measure improvement in student test scores year over year and attribute this difference to the teacher.
  • These are highly stochastic—errors from two statistical guesses are compounded.
    • Jordan Weissman: “…when you've got two numbers and looking at the difference, each of those numbers has some randomness and variability and uncertainty… it just compounds all of that.” [10:26]
    • Cathy O’Neil: “...the idea that we’re going to sort of hold each teacher accountable for this sort of average error term for their class is ridiculous. Especially because you typically only have about 20 kids in the class.” [10:53]

• Negative Side Effects:

  • Incentivizes cheating and harms innocent teachers (Washington D.C. example under Michelle Rhee).
  • Good teachers flee systems with these capricious models—leads to poorer education for disadvantaged schools.
  • Felix Salmon: “...all you're doing is you're feeling that you're living in this kind of incredibly unfair lottery system.” [14:43]

• Lack of Self-Correction:

  • Unlike baseball statistics, these models don't learn or improve; no feedback loop.
    • Jordan Weissman: “[Sports teams] can look and see, okay, there are errors. Let's reincorporate that and improve. [...] You talk about how that doesn't exist in the value-added model.” [15:32]

---

3. Algorithms with Harmful Side Effects: Recidivism Risk Models

[17:00–26:12]

• Recidivism Algorithms in Criminal Justice:

  • Used by judges in over half the U.S. states for bail, parole, and sentencing decisions.
  • These models often reinforce racial and economic bias because historical data reflects uneven, biased policing.
    • Cathy O’Neil: “...proxies for race and class…there’s just like the data itself going into these algorithms…if we think...it is uneven, unfair, biased and racist, then these algorithms are just going to further those biases.” [23:27]

• Misuse vs. Potential for Good:

  • The same models could be used to offer help/interventions instead of harsher sentences.
    • O’Neil: “If they were used as ways of finding an intervention that actually reduced recidivism…then it wouldn’t be a weapon of math destruction…” [22:06]

• Ethical Tradeoff: Efficiency vs. Fairness

  • Jordan Weissman: “Are we willing to sacrifice some efficiency for fairness?...this is not something that’s foreign to American thinking.” [25:26]

• Opacity and Accountability:

  • Data and algorithms are proprietary—researchers and the public can’t audit them.
    • Felix Salmon: “...the proprietary nature of these algorithms…People look at them and say, why is this secret? And there's no good answer.” [27:18]

---

4. Solutions: Regulation, Professional Ethics, and the European Model

[27:46–34:42]

• Professional Ethics:

  • Could data scientists adopt a version of the Hippocratic Oath? O’Neil argues self-regulation is not enough; companies face profit motives that conflict with fairness.

• Regulation and Oversight:

  • The European approach: strict rules on data reuse/profiling (GDPR-style).
    • Jordan Weissman: “...the regulatory state saying you cannot take people's data and then resell it without their permission...That seemed like the most elegant solution.” [29:45]
    • O’Neil agrees it’s powerful but politically challenging and notes it wouldn’t fix all issues (e.g., teacher and sentencing algorithms).

• Call for a New Regulator:

  • O’Neil proposes a regulatory body to oversee high-impact, high-secrecy algorithms — especially those affecting large populations and where harm is possible.
    • Cathy O'Neil: “...when it becomes widespread and high impact, then it can't be secret and potentially destructive…there has to be at least a regulatory body that can look into it to make sure it’s not discriminated.” [31:07]

• Due Process Concerns:

  • Could denial of algorithmic transparency itself violate constitutional rights (due process)?
    • Jordan Weissman: “If an algorithm is opaque...you're not getting due process of law.” [31:44]

• Current State of U.S. Protection:

  • The Consumer Financial Protection Bureau (CFPB) is attempting to analyze disparate impacts, especially in loans.
    • Felix Salmon: “[CFPB is] doing quite a good job at trying to find what's known as disparate impact...where you have an algorithm which has a disproportionate impact on black people...” [34:02]

---

5. Broader Societal Impact and the Digital Underclass

[34:12–35:29]

• Algorithms exacerbate inequality:
  • They often hurt the poor and people of color, while the affluent rarely encounter their worst effects.
    • Cathy O’Neil: “...well-off people do not even have to look at these WMDs.” [35:13]
  • The commodification of user data and profiling targets vulnerable groups for predatory services.

---

Notable Quotes

• Cathy O’Neil: “So often you just hear people sort of vaguely talking about, you know, ills that could befall humanity, and I’m like, what are we worried about? Can we define it? Can we triage, can we carve it out?” [03:31]
• Felix Salmon: “The broad effect is that the good teachers wanting to get out from under this sword all wind up going to the good school systems where they don’t need to worry.” [14:43]
• Jordan Weissman: “Are we willing to sacrifice some efficiency for fairness? ...Maybe the most important policy question.” [25:26]
• Cathy O’Neil: “What I'd object to is the way these models are used to destroy people's lives.” [22:06]
• Felix Salmon: “...the book, the main worry...is that while rich and affluent people might actually be better off from a lot of these algorithms, it's...the poor, the people of color who really get hurt and who are largely voiceless and have no ability to fight back against these models which are punishing them.” [34:12]
• Cathy O’Neil: “It will never happen that all algorithms have suddenly become transparent. ...But what we may be able to say is, like, when it becomes widespread and high impact, then it can’t be secret...“ [31:07]

---

Numbers Round (Memorable Moments)

[35:31–end]

• $185 million: Penalty Wells Fargo pays for fake accounts scandal — an example of how measurement and incentive systems can go awry. [35:43–37:47]
• $182 million: Facebook’s annual Norwegian ad revenue, compared to the controversy over Facebook’s content policies and international influence. [38:46–40:46]
• 16: Percentage that black-sounding names are more likely to be rejected by Airbnb hosts — a stark example of algorithm-enabled, or at least amplified, discrimination. [41:01–41:29]
• 99: The book’s peak ranking on Amazon’s overall bestseller chart—testament to the impact and timeliness of these issues. [42:23]

---

Episode Flow & Tone

The episode is casual, nerdy, and often humorous, but remains focused on unpacking complex ethical and technical dilemmas. Felix Salmon keeps the flow lively, O’Neil brings technical rigor with real-world urgency, and Weissman provides relatable audience reactions and policy perspective. The group stresses empathy for those adversely affected, and a sense of urgency to address these silent but pervasive forces.

---

Timestamps for Key Segments

• 04:20–08:00: Definitions—What is an algorithm? What makes one a WMD?
• 08:18–17:00: Teacher value-added models—bad algorithm deep dive.
• 17:00–26:12: Recidivism risk models—harmful effects of algorithms that “work.”
• 27:46–34:42: Regulatory solutions; ethical data science; European approaches.
• 35:31–end: Numbers round—recent news stories as seen through the WMD lens.

---

Conclusion

The episode makes a compelling case for treating algorithmic decision-making as a major social and civil rights issue. In O’Neil’s words and analysis, the hosts show how bad or misapplied data science can reinforce and deepen social inequities, while outlining what it might mean for data science to truly serve the public good. The conversation ties together technical, ethical, and political threads—offering listeners not only a warning, but a framework for policy and resistance.

Transcript

[Jordan Weissman]

The following podcast contains explicit language.

[Felix Salmon]

Hello, and welcome to the extra special Amazing Weapons of Mass Destruction edition of Slate Money. We three brave souls have now managed, between the three of us, to publish one book.

[Cathy O'Neil]

Wow.

[Felix Salmon]

And.

[Jordan Weissman]

But it's a really good book.

[Felix Salmon]

It's a really good book.

[Jordan Weissman]

It's a really, really good book.

[Felix Salmon]

I am Felix Ham and the Fusion. I have never published a book. I am joined by Jordan Weissman of Slate, who has never published a book.

[Jordan Weissman]

Probably never going to.

[Felix Salmon]

We are also joined by Kathy o', Neill, who has published a book. It is called. What is the title?

[Cathy O'Neil]

It's called Weapons of Math. How Big Data Increases Inequality and Threatens Democracy.

[Felix Salmon]

And we are going to devote this entire episode to this book because we love Kathy and we are basically, we're taking book plugs to a whole new level here.

[Jordan Weissman]

I was gonna say we bring in all sorts of other people to plug their books. It would be a little bit impolite if we told Kathy she couldn't.

[Felix Salmon]

At the end of this episode, you should go out and order multiple copies and send them to everyone who you think should read it. And at the end of this episode, you will understand why so many people should read it.

[Cathy O'Neil]

And if I may plug. I just wanna say, since we were a podcast, that I read this book on the audio version of this book. So feel free to buy that as well.

[Felix Salmon]

If you like the Kathy voice.

[Cathy O'Neil]

Some people love it.

[Jordan Weissman]

Mellifluous. Kathy o'. Neal.

[Felix Salmon]

If you like the Kathy voice, you can audio listen instead of read. Some people. Yeah. It's much easier when you're driving.

[Jordan Weissman]

Yeah.

[Cathy O'Neil]

People.

[Jordan Weissman]

Yeah, it's true.

[Felix Salmon]

So we are gonna break this episode into the standard three parts. The main story of the book, as I'm sure all regular readers or listeners of Slate Money will know, is that there are these algorithms which are quietly destroying society. And so what we are going to do is we're going to spend the first part talking about some really bad algorithms. One in particular, we're going to spend the second part talking about algorithms which are actually quite well designed and efficient, but they have very nasty side effects, you might say. And then we're going to finish off in the third part by talking about what we can do about this, because this is not an entirely pessimistic book. It's just a mostly pessimistic book. So we're going to. I'm going to start.

[Cathy O'Neil]

It's a manifesto.

[Jordan Weissman]

Can I actually say one thing really quickly at the start of why I really, having podcasted with you for, like, two. Two Years. More than two years now. And still really loved reading this was.

[Cathy O'Neil]

Please do, Jordan.

[Jordan Weissman]

No, I really do. I really.

[Felix Salmon]

We need some more. Cassie Flathria.

[Jordan Weissman]

No, I really do. So I typically subscribe to the gore of Idal thing. When one of my friends does, like, succeeds, a little part of me dies. But, like, reading this is just, like, so, like, I don't. I'm not.

[Cathy O'Neil]

So you're like, I'm gonna hate on copy.

[Jordan Weissman]

But I was reading this, and it's just like, this is so, like, it's so good because it just so clearly articulates so many of the themes you've been kind of laying out for a long time, and it formalizes them. You give a great model for when an algorithm is going to malfunction and become this, like, weapon of math destruction.

[Felix Salmon]

It is a tripartite test.

[Jordan Weissman]

Yeah. And that's what's great about it. It gives you a way to think about looking at an algorithm and when is it going to turn into this. This terrible thing, the weapon of math.

[Cathy O'Neil]

Destruction, you know, and thank you for saying that, because that was one of my main goals, is to perform a triage on this conversation. Because so often you just hear people sort of vaguely talking about, you know, ills that could befall humanity, and I'm like, what are we worried about? Like, can we define it? Can we, like, triage? Can we carve it out?

[Jordan Weissman]

And I remember when we first started talking about the stuff on the show, Felix and I were sitting here going, like, why is this stuff a big deal? And you kind of slowly started converting us. And now I feel like I know not only why is this a big deal? But, like, how to think about it and how to really worry about being a big deal. And I'm gonna stop now, just heaping praise on you instead, let you talk about what's going on in this book. But I just wanted to really put that out there.

[Cathy O'Neil]

Thank you.

[Felix Salmon]

Right, so let's. Let's start. Actually, why don't we start? Because this is a nerd podcast for nerds.

[Cathy O'Neil]

Yeah.

[Felix Salmon]

I feel like it's okay to start with two little sort of nerdy stipulations so we can start off with two definitions. Number one is, what is an algorithm? And number two is when is an algorithm a weapon of math destruction?

[Cathy O'Neil]

Great. Okay, so what's an algorithm? And obviously, I'm not going to go into a lot of detail here, but I do want to say that the main two things you need when you are building an algorithm is data. And that's it's called training data. You're training an algorithm to sort of look for patterns. And then the second thing is a definition of success, also known as an objective function, something some target, you're saying, hey, pay attention when this happens. So it could be like, pay attention when someone goes to jail, pay attention when somebody is successful at a job, or something like that. And then what the algorithm does is it works, looks for patterns in the history that you're giving it in the data for when this definition of success actually occurs.

[Felix Salmon]

So if you know those of you out there who are investors who like to buy value stocks, and you look at price to earnings ratios, and you buy stocks which are trading on low PEs because you think that means they're undervalued and they're going to go up when they become fairly valued, what you're basically doing is you're applying a model there which says if you buy stocks when they're cheap and then sell them when they're expensive, you wind up making money.

[Cathy O'Neil]

Exactly. But the only thing I would add is that you have to be very precise when you build a model about what you mean by will go up. Like you have to say, within a certain time period. You know, by this, you know, you have to actually define all your terms. But yes, that's basically what you're doing.

[Felix Salmon]

And obviously the financial markets are full of battling bots and we can let them battle each other to a standstill and it doesn't really harm most of us. Unless you're Michael Lewis, in which case it benefits you greatly because you get a massive book advance. But there are much more dangerous and invidious or insidious algorithms out there, which you call weapons of mass destruction. And you have this wonderful three part test. Can you tell us what are the three criteria that, that an algorithm needs in order to be a wmd?

[Cathy O'Neil]

Right. So just to be clear, I am not a hater, I'm a lover. Most algorithms are fine. They're benign. I don't care about them. Most people don't care about the algorithms I build. But sometimes they are important and destructive. And so here's my three characteristics. The first is that they're widespread and they have high impact. That means that the result of this algorithm is somehow used to make decisions that affect people's lives. Like it might send them to jail, it might not let them have a job or not have a job. Makes important decisions about people. A lot of people. The second is that it's secret in some sense. So that often it's often a scoring algorithm, and it's secret in this, in a sense, in the context of a scoring algorithm means that people do not understand the formula they're being scored by. And sometimes they don't even understand that they're being scored. It's actually, like, away from them, they don't even see it. And along with secrecy, you have this sort of pairing that it's typically unaccountable and you can't appeal. So if you don't know you're being scored, you can't say, hey, that's the wrong score. But even if you do know your score, if you're not told how it works, you can't complain and you can't correct it. The final characteristic is that it's destructive. And that's in2 at two levels. The first is that, you know, like the person. The people that are scored badly suffer, right? They are. They lose opportunities in their life, important opportunities. But the second, which I see as a pattern for all these weapons of mass destruction, is that they are. They're engendering these sort of negative feedback loops that actually undermine the original goals, which are often good goals, good intentions. But, like, you have these good intentions, you have these bad algorithms that are trying to fix these big problems, and then they're actually making the problems worse.

[Felix Salmon]

All right, so let's start with a bad algorithm. Just a simple. An algorithm which is badly designed. It almost doesn't count as an algorithm because as you said, one of the things that you need for an algorithm is a success metric. And you have to be able to tell whether it's working or not. And what we're going to talk about is these exist in various different places. They certainly exist in Washington, D.C. where your main anecdote comes from, but they exist in New York and more or less across the country, teacher rating algorithms, where what you do is you try and measure how much a class improves over the course of a year, and then you attribute that improvement to the teacher of the class. And then you say that a good teacher will make a class of kids improve a lot and a bad teacher will see no improvement in their kids. And then what you do invariably is you use these ways to sort of punish in some way or even sometimes fire the teachers who perform badly.

[Cathy O'Neil]

Yeah, so it's called value added model. And actually it's a family of models. Different models are different versions of the model, sometimes called a growth model. Those are the same things. And it relies, as you say, on this kind of expectation. And it's actually another model. So for each person in each class, for each teacher, they're given a sort of expected end of year standardized test score. Okay, so let's call that the first model. And then the teacher is basically held responsible for the difference between that expected score and the actual score for their class. And you know, think about it like if you understand statistics at all, and I know some listeners do, this is called the error term. In that first model. It is literally the difference between what was actually happened and what was expected to happen is the error term.

[Felix Salmon]

In other words, you are not looking at the number. You are looking at the difference between two numbers. And whenever you look at the difference between two numbers, you are going to look at something which is naturally much fuzzier and more stochastic than the actual.

[Jordan Weissman]

Main big number to make, I guess, a slightly dumbed down version of that point. When you've got two numbers and looking at the difference, each of those numbers has sort of some, some randomness and variability and uncertainty built into it. And so you look at the difference between those two slightly random, slightly uncertain numbers and it just compounds all of that.

[Cathy O'Neil]

Absolutely.

[Jordan Weissman]

And so that's what makes it such like that kind of second, that derivative of it.

[Cathy O'Neil]

It's a derivative, actually. It is actually called a derivative.

[Felix Salmon]

It's the first derivative.

[Cathy O'Neil]

So y. And moreover, I'll add that like the first model I mentioned that tried to guess what a kid is going to get as a score, knowing what they got last year. Essentially it's a bad model already. It's already very uncertain, as you point out, Jordan. And the idea that we're going to sort of hold each teacher accountable for this sort of average error term for their class is ridiculous. Especially because you typically only have about 20 kids in the class. I mean, it'd be one thing if like you did this over many, many thousands of years and you had a million and a half students and, and then you probably would get a signal, but you don't. That's not what we have.

[Jordan Weissman]

This is one of the points you made that I really appreciated, which is that just, you know, we like to think that we can bring the statistical certainty to measuring something like teacher quality. But statistics inherently, as a science deals with big numbers. That's how you even out the variability. And there are just some things where we can't get that kind of scale. And I think of, you know, there's this like really famous economic study about the value of a good teacher and says that a value added teacher will add this much to a kid's lifetime earnings. And it's been a while since I've looked into it, but as I was reading your piece or your book, I was thinking about the difference between that study which just looked at mounds of data mounds and mounds of data, versus how you're actually rating a real teacher in the classroom based on 125 kid class, and just what a gap that is in terms of, you know, the academic work, looking at a data set and what would actually be done on the ground.

[Cathy O'Neil]

Yeah.

[Jordan Weissman]

And I think, you know, really you have to keep that in mind when you think here, people whipping out these studies to try and justify then these rating systems. Yeah, it was.

[Cathy O'Neil]

Right. Thanks. Well said. And I just want to add that like on top of the statistical uncertainty, like what I looked at in Washington D.C. which happened like around 2010, was Michelle Rhee was chancellor of schools there and she added something which, you know, she should have seen coming. Right, Everybody should have seen coming. But she added like carrots and sticks to the system. So she had all the teachers evaluated this way with these faulty models. And then she also said, you're going to get a bonus if you get a good score and you're going to get fired if you get a bad score. And so I actually interviewed someone who got fired by it. But I mean, just, just stop right there and think about what that would look like if you're going to get a bonus if you get a good score, guess what's going to happen. You're going to make sure that your kids do well on their standardized tests. So we actually saw a bunch of allegations of cheating. And then, you know, they were never really, really investigated because it was so embarrassing politically. But then some of those kids whose teachers were probably cheating were then sent to this woman, Sarah's class. And guess what? Their expected score at the end of Sarah's year was higher than what they actually achieved. So she got a very bad score. Right. So previous cheating actually ruins the next person's.

[Felix Salmon]

And there's one other thing which we have to talk about here, which is the meta effect of all this, which is that if you are a teacher in a school system, which is, which has a system, a quantitative system of weeding out so called bad teachers. And if you are, and if as we have seen certainly in New York and even in Washington as well, if the bad scores tend to be distributed more or less randomly, that there's very, very little correlation between teacher quality and having a bad score, then what you're doing is you're feeling that you're living in this kind of incredibly unfair lottery system. And there are lots of school systems in America. There are lots of schools in America and some of them have teachers who are suffering under this uncertainty and some of them don't. And if you're a decent teacher who can get a job in more or less any school system, chances are you're going to go to one of the more affluent school systems who don't have this, you know, sword of Damocles hanging over the head of every teacher. And so the broad effect is that the good teachers wanting to get out from under this sword all wind up going to the good school systems where they don't need to worry.

[Cathy O'Neil]

And moreover, the rich school systems which don't use these tests.

[Jordan Weissman]

This actually leads to one other really, I think, important point from this section of the book that I want to bring up because again, I found it really enlightening, which is just these teacher, these value added models, they're not self correcting. You have this kind of comparison where you talk about the way we rate teachers versus how a sports team will rate players. And they, you know, baseball team has its models for like a pitcher. Right. And they run the potential prospect through that model. And if they say, oh, we don't want that guy, they can then look. And if that guy they decide not to draft turns into a superstar, they can say, what did we get wrong in our model?

[Cathy O'Neil]

What do we get wrong?

[Jordan Weissman]

What did we get wrong? And we can fix it. They can look and see, okay, well there are errors. Let's reincorporate that and improve. And, and you know, you talk about how there isn't. That really doesn't exist in the value added model. I might argue that if like test scores don't start going up, maybe theoretically a school district would try to start improving it. But I don't know of how many.

[Felix Salmon]

Instances where there's no, there's no way you can do the feedback loop because you have no data. Like when you know, teacher Sarah goes to another school district and starts performing very well, that there's no way of knowing that you're getting that data and building it back into the algorithm.

[Cathy O'Neil]

Exactly. It's not individually, there's no individual feedback, there's no ground truth.

[Jordan Weissman]

Yeah.

[Cathy O'Neil]

It's only as a system we can say, hey, this isn't working. And I think we might be starting to see that, like we're not actually suddenly teaching all our kids really well.

[Felix Salmon]

And not to mention the fact that there doesn't seem to be any correlation whatsoever between school systems which have these kind of rankings and school systems which perform well even on standardized tests. There's no particular reason to believe that test scores which are obviously not the be all and end all of education, but even assuming that they are, the test scores respond well to this kind of system.

[Jordan Weissman]

Just speaking to like your, the way you think about an algorithm generally, this point about being able to self correct that an algorithm can learn and if there's no way for it to really learn, then it's a bad algorithm. I thought that was a really. That's a point I'm going to take away from, from this. I don't want to ruin. I don't want to spoil too much of the buck.

[Felix Salmon]

No, but that's so. Okay, so that's the bad algorithm. So Kathy. Yeah, There are bad algorithms like teachers and then there are algorithms which actually do what they're meant to do. Yeah, so talk about one of those.

[Cathy O'Neil]

Yeah, I'll talk about one of them. There's quite a few examples in my book that are successful from the point of view of a certain person, but are actually in my opinion bad for society.

[Jordan Weissman]

Nefarious.

[Felix Salmon]

They can make money for a company, but they're bad for society.

[Cathy O'Neil]

Right. So yeah, very, very shortly. The scheduling algorithms that keep people like on call for Walmart is an example of something that's like makes probably saves a lot of money. Like the shift workers don't, but the shift workers like quality of life is degraded. But that's not the example we're going to talk about. We're going to talk about recidivism risk models, which are models that rate people sort of like score people on their chances of coming back to prison after leaving. So 97% of people who go to prison eventually leave. So this is a very salient question, like who's coming back? So there's a lot of stuff here, but the first thing to understand is that judges use these scores in like more than half the states for all sorts of decisions including parole, bail and sentencing. But I'm going to focus on sentencing because I think it's the most scary one. And the idea is like people who have a high score of coming back to jail will be sentenced to longer.

[Felix Salmon]

And what is the. I mean before we go into the harmful effects of the algorithm, whether the algorithm is good or bad, can you just explain the thought process whereby, I mean. Okay, let me hazard a guess here.

[Cathy O'Neil]

Yeah, yeah.

[Felix Salmon]

That the I'm hazardous that basically one of the main purposes of Prison is to keep criminals off the street. And if you're in prison, then you're not going to be criming out in public. And if you, if you are very unlikely to crime again upon release from prison, we might as well release you because there's no point to society in just keeping you locked up indefinitely. We. Whereas if you are very likely to crime again, then it helps society to keep you in prison where you're not gonna do any harm. And for that reason, if you have a higher chance of reoffending, then it makes sense to pass to hand down a higher sentence for exactly the same crime.

[Cathy O'Neil]

Right? And I think that is the reasoning. Now, I'm not a lawyer, but I know that judges are given a long list like a rubric for deciding sentencing. And it's complicated, it's not simple. And it's partly protecting the public, it's partly just desser is partly all this stuff. Right? But yes, I think the reasoning is more or less like that. I do want to throw in to complicate this because it really is complicated that we really need to start distinguishing types of crime. A lot of crime is stuff that we really wouldn't want people out criming again, you know, murder, rape, you know, assault. And then there's a lot, a lot of crime that is stuff like basically mental health problems that get people arrested. And that happens a lot. So if you have like a drug, you know, drug addictions and these people will tend to have very high recidivism risk scores. But we don't think of them as like really, really evil people. We think of them as like people who need treatment. Right. But sort of that comes to the very first point, which is like we are, we're not sort of being sensitive to the question of why.

[Felix Salmon]

And I'm going to jump in here before we return to recidivism risk and talk about one little baby anecdote from the book, which I think is germane here, which is what happens when universities realize that a certain subset of their incoming freshmen are unlikely to finish their freshman year and they're going to drop out and they're basically going to fail certain universities and we're not going to start naming names, are going to basically kick those freshmen out sharpish because if they aren't even really counted as part of the freshman class, then that will improve their U.S. news report using U.S. news and World Report rankings. Other universities will look at those people and say, ah, you are the ones who need the most help and you'll get Extra, like buddies and, and, you know, teaching. And basically we're going to do our very best to make sure that you finish. And I feel like that distinction where you can have, like, a relatively sort of morally neutral algorithm which can either have a very evil or a very good outcome is something which you can definitely apply to people with mental health issues or drug addictions in the criminal justice system, where you. Whereby, on the one hand, you can say, oh, you're going to need, like, extra help and we can help, like, fix the underlying issue here. Or you can just say, hey, you know what? We're going to slap an extra five years on your sentence.

[Cathy O'Neil]

That's such a good point. Thank you, Felix. Like, I actually don't object to the existence of these models and we're going to talk about, like, all the data that goes into it and how, like, I think it's very racially biased, these scores. But the main point is that if it's. It's. What I'd object to is the way they are used to destroy people's lives. Right. If they were used as ways of finding an intervention that actually reduced recidivism, like, do you need counseling? Do you need mental health facilities? Then it wouldn't be a weapon of mass destruction because it wouldn't have the third characteristic, which is destructive. It would actually be proactive and it would, like, help people who are in trouble.

[Jordan Weissman]

Yeah, it's like each one of these, you know, every evil algorithm now that you talk about has a possible use for good. Because essentially what you say is that we've gotten excellent at finding troubled people. Like companies that are selling predatory products are looking for troubled people. And if you can find those troubled people, you can either choose to sell them a payday loan, give them a payday loan, or you can try to help them figure out their finances.

[Cathy O'Neil]

That's a really good way of saying.

[Jordan Weissman]

Every, you know, we've learned how to find society's weakest, and now it's a question of, you know, with great power comes great responsibility. Do we think about this like, you know, Uncle Ben?

[Felix Salmon]

So we're gonna, we're gonna come back to this in the solutions bit, but let's just, like, explain what's going on in these. Recidivism.

[Cathy O'Neil]

Yeah, right. So I just want to say, and a lot of listeners already know this, but the data going into these algorithms, which, you know, basically is crime data, and there's also questionnaires associated to these recidivism risk algorithms. They're. They're all basically proxies for race and class. And the reason is that we have a sort of uneven policing system in our country. So we just, people get, especially with low level nuisance crimes and mental health problems, which I just, just discussed, you're just much more likely to get caught for something like that if you are black and poor. So there's just like the data itself going into these algorithms. And remember algorithms, all they do is train, train on historical patterns to an objective function which in this case is going to jail. And if so, in other words, if we think that the way we send people to jail is, is uneven, unfair, biased and racist, then these algorithms are just going to further those biases. And that's what's actually happening. And talking about toxic overall feedback loops, these algorithms were introduced to the justice system because we wanted to improve, make justice more scientific and objective and improve racist practices.

[Felix Salmon]

Because no one is denying that, that sentencing in general is racist.

[Cathy O'Neil]

Yeah, nobody. Nobody. There's plenty of evidence for that. So this, the idea was like, let's do better than that. And it's a good idea. But the problem is that what we're doing, since we're being sloppy about it, very sloppy, is we are using these algorithms to these scoring systems to hold people accountable yet again for things like they were born poor and black.

[Jordan Weissman]

And so this is a point you bring up and you kind of ask the reader to think hard about this one really important policy question, which is are we willing to sacrifice some efficiency for fairness? Because this sentencing algorithm is probably pretty efficient. Being racist can be efficient in some ways.

[Cathy O'Neil]

It's actually, it's incredibly, if you think about it, it's accurate because yes, black people are more likely to go back in jail because guess what, they're more likely to get caught again for the same.

[Jordan Weissman]

Exactly. And so it's absolutely. So the question is, are we willing to sacrifice some efficiency for fairness? Which is theoretically what our, you know, you point out that's kind of what like the fourth Amendment with the whole Constitution is about in a lot of ways. Yes. This is not something that's, you know, that's foreign to American thinking. It's, you know, we are supposed to sacrifice some efficiency in the system to make this.

[Felix Salmon]

And you have a great chapter about all of the efficient but harmful laws which, you know, practices which were outlawed because, you know, the harm outweighs the efficiency.

[Cathy O'Neil]

Yeah. I mean, in this case it's actually constitutional. Right. Like you're supposed to have be equal in front of the law.

[Felix Salmon]

So let me.

[Cathy O'Neil]

Justice issue.

[Felix Salmon]

So let Me ask the, the obvious question here, which is, do you have any reason to believe that racist. As though they almost certainly are, that these algorithms do not constitute an improvement on the status quo ante? I mean, I think what a lot of people who implement these algorithms believe and honestly believe is that the sentencing system is so institutionally racist to begin with, that even a slightly racist algorithm is going to be an improvement.

[Cathy O'Neil]

I would like nothing more, Felix, than to be allowed to. To check. That is one of my life's goals, to have the access to the data to test. And it's possible. Now we have, like, jurisdictions that use these risk recidivism scores, and we have jurisdictions that don't. So there's sort of a natural experiment to see whether it's become less racist as they use these scores. So bring it on. The problem is that data is not.

[Felix Salmon]

Accessible to me, and this is, and this is a theme which runs through the whole book, is the proprietary nature of these algorithms. And time and time again, people look at them and say, why is this secret? And there's no good answer.

[Jordan Weissman]

There's no good answer other than it's our secret sauce.

[Cathy O'Neil]

That's the marketing.

[Jordan Weissman]

Yeah. The company. This is our secret sauce. Hey, you can't tell them what the spices are.

[Felix Salmon]

So, Jordan.

[Jordan Weissman]

Yes.

[Felix Salmon]

Every book.

[Jordan Weissman]

Yeah.

[Felix Salmon]

Which on. I would say at least nine out of ten books that we talk about on this show has. Has a solutions chapter. And in a way, Kathy's book is no different. It has this conclusion. It has this conclusion where she. Where she talks about how data scientists have the ability to inflict just as much harm as doctors do. But doctors have a Hippocratic oath, and shouldn't data scientists have something similar? What did you make of this conclusion?

[Jordan Weissman]

Well, I mean, okay, so I go Kathy a. More. She starts off with the Hippocratic oath and saying that's actually not enough. In a way. You know, you're. You can't rely. I mean, you can try to inculcate some responsibility in data scientists, but in the end, company, you can't rely on the free market to police itself here. Because in the end, it's just. The example you give is that you can rely on the free market to improve on things like social issues, like companies will embrace gay rights because it's profitable. But can you really expect a company to improve on something like civil, on fairness when it's less profitable, less efficient for them to be fair and to kind of tone down these algorithms? You know, you talk about a few different things and you know the one that stuck with me is sort of, you know, kind of the most blunt and hardest to do. But you talk about kind of the European solution towards the end, which is really like. It is the. It's the. It's the cleaver. It's basically saying to companies that you can't reuse. You can't reuse and resell my data. Like, Right. The regulatory state saying you cannot take people's data and then resell it without their permission. It is just not allowed. You end up cutting off a lot of these data brokers which create these profiles that can end up preventing you from maybe getting a job or from, you know, getting a loan at some point in your life and just say no more. And that, you know, that seemed like the most elegant solution. I'm wondering, though, you know, do you think that to me, seems like the most elegant solution. I'm drawn to it. I'm wondering if you think some of the kind of less optimal, less elegant solutions could still work.

[Cathy O'Neil]

So, yeah, I mean, we didn't actually talk about any of the examples that use, like, profiles of people. And some of them are really important.

[Jordan Weissman]

You gotta read the book for that.

[Cathy O'Neil]

Political micro targeting, you know, like predatory advertising, as you said, like people trying to get jobs, but they're checked out with their sort of online behavior. And I agree that the European model, which basically gets rid of data warehousing and profiling for the most part, is the big gun. I have to say, though, that, like, other authors have suggested something along these lines and have been, like, absolutely panned. I mean, like, the political pushback to this, it's really. You're talking about, like, getting rid of a large industry. Yeah, I'm not saying that's. That we should say, hey, that no problem. I'm just saying politically, that's really difficult. And I would also add that it doesn't solve the problems for the teachers.

[Jordan Weissman]

That's true.

[Cathy O'Neil]

It doesn't solve the problems for the criminal defendants. So is not the only thing that we could do. And I would love to see it happen. But I just.

[Felix Salmon]

But you. What you do suggest is that we create basically a whole new regulatory body which is in charge of keeping an eye on these algorithms and these weapons of mass destruction and has. And which has some kind of power to stop them if they. If, you know, if they start.

[Cathy O'Neil]

Right. And I just. I just want to go back to the very beginning. You know, we talked about how I've carved out this definition of what to worry about. And I think we just need to start there when we're talking about regulation, because we need it will. It will never happen that all algorithms have suddenly become transparent. Right. That's not going to happen. I don't want that to happen. It would be ridiculous. What we. But what we may be able to say is, like, when it becomes widespread and high impact, then it can't be secret and potentially destructive. Then it can't be like entirely secret. Like, there has to be at least a regulatory body that can look into it to make sure it's not discriminated.

[Jordan Weissman]

You know, also, the law nerd in me is sort of. The wheels of my mind are turning. If there isn't some sort of due process argument that could be made in the courts about a lot of this stuff where these algorithms start intersecting with things like sentencing or, you know, an employer, like, you know, a state, a teacher, you know, being fired, essentially saying that if an algorithm is opaque, if it's a black box and you can't know what you're being rated on, you're not getting due process of law. I do wonder.

[Felix Salmon]

This is something which I've been wondering about recently. I've noticed this hasn't happened for everyone, but it's happened for a lot of people that when you call an Uber X now, instead of just being charged by the mile in a minute, it just gives you a flat fare. So if you're going here and it's going to cost you $14.27 and then you go, okay, and then you go there and then you get charged $14.27. And that algorithm for how much it's going to cost you is completely opaque and can easily be a function of how I've been profiled according to my. What's the word? Elasticity of like, they. They want to charge me basically as much as I would be willing to pay to do. Else taking the same trip might get charged less because they wouldn't be willing to pay as much as me. And there's a lot of things going on.

[Cathy O'Neil]

Insurance, you see that?

[Felix Salmon]

Insurance don't know.

[Cathy O'Neil]

Yeah, yeah, absolutely. So when it rises to the level of harm, like real potential harm, that I really feel like we need regulation. We need someone, maybe not everyone, but someone to be able to check this shit out. Right now we have nothing.

[Jordan Weissman]

Yeah, Pricing models need to be. It's, you know, I mean, some of this seems like the kind of thing that Consumer Protection Financial Bureau would be able to do.

[Cathy O'Neil]

I tried to talk to them about this. The problem is that they, you know, and I'm not singling them out in any sense, but like regulators are like they're afraid of technology.

[Jordan Weissman]

I feel like if CFPB has data.

[Felix Salmon]

Scientists, it's not like, it's not like they're afraid of technology. But the fact is, and they want.

[Cathy O'Neil]

To work on this, they just don't. They're not there yet is all.

[Jordan Weissman]

I'm just dropped copies of this book on their doorstep.

[Cathy O'Neil]

Hey, listen, actually the cfpb, I should say that like they've done the most work in this area. Like they, well they didn't do. They weren't like analyzing like auditing an algorithm per se, but they audited sort of auto loans, predatory auto loans. And they. And so it's been a kind of political mess, but they're really trying to do this.

[Felix Salmon]

They're doing quite a good job at trying to find what's known as disparate impact.

[Cathy O'Neil]

Yes.

[Felix Salmon]

Where you have an algorithm which has a disproportionate impact on black people, basically.

[Cathy O'Neil]

Well, that's a lot of a large part of the album.

[Felix Salmon]

That's a large part of the book. And to be honest, you know, the book, the main worry of the book is that while rich and affluent people might actually be better off from a lot of these algorithms, it's the 99% and actually more like the bottom 40%, the poor, the people of color who really get hurt and who are largely voiceless and have no ability to fight back against these models which are punishing them.

[Cathy O'Neil]

Yeah. And they, and they are the underclass sort of digitally speaking. And moreover we technologists and this. Actually I'm going back to the sort of origin story of the book. Like the moment I realized I needed to write this book was when I was like hanging out with VCs in my technical startup and they were like the VC was talking about how great it's going to be someday when all he sees are trips to Aruba and jet skis in his tailored ads and he never has to look at another University of Phoenix ad because it's not for people like him.

[Jordan Weissman]

And meanwhile the predatory universities are going to be sitting there and you know, Joe down the street is going to be getting nothing but.

[Cathy O'Neil]

So in other words, sort of like the Internet technology is set up so that well off people do not even have to look at these WMDs.

[Felix Salmon]

So let's have a numbers round here.

[Cathy O'Neil]

Yeah, let's do it.

[Felix Salmon]

Let's do a numbers round. Jordan is gonna go first because he's convinced that we all have his number I'm pretty sure I don't have his number.

[Cathy O'Neil]

I know what his number is, but I don't have it. I have two numbers and I'm like, Jordan's gonna take the first one.

[Felix Salmon]

Jordan, what's your number?

[Jordan Weissman]

My number is 185 million.

[Cathy O'Neil]

I knew it.

[Jordan Weissman]

Which is how many dollars Wells Fargo is having to pay in various funds, mostly to the cfpb, Los Angeles, City of Los Angeles.

[Felix Salmon]

I kind of love.

[Jordan Weissman]

Yeah, for turns out they were, their employees were opening thousands, hundreds of thousands of fake accounts and credit cards and such for customers basically to fill quotas there. They had this system where they were basically told you were going to be judged on how many accounts do you open every day or how many cards you get people to open. And so what did people do? And they had these incentives. Well, they just went, didn't tell anyone, didn't tell the customers they were doing it. They would just open something and use information, open some account with like no money in it or a dollar in it maybe, and then move that money back into the customer's account often. But just to have opened it, just to add it to their quota, they.

[Felix Salmon]

Would sign people up for credit. They would say, well, if you want to open up this savings account, you need to open up a credit card. At the same time, get the credit card at home, it has no annual fee. You can just cut it up. This is clearly the opposite of profitable for Wells Fargo. So people got really excited about the fact that some of these checking accounts and some of these credit cards had fees and that people wound up having to pay fees on these accounts, which they never asked to open. The fact is that the fees were a rounding error. Like, the total number of fees generated by all of the credit cards was like $400,000.

[Cathy O'Neil]

I'm going to push back here. I want you to look at this as a regime that the management placed on all these customer facing people who make basically $10 an hour. You can't say the fraud didn't make them money without talking about the overall, like, system of incentives. Like, the question is whether putting the quotas into place as a whole was profitable. And I think it probably was, because not, I mean, there was fraud, clearly, but there was also probably a lot of, like, very pushy salespeople that succeeded in getting people to open accounts, maybe.

[Felix Salmon]

And again, this comes back to the overarching question of algorithms, which is like, does, you know, all algorithms get built with a model in mind?

[Cathy O'Neil]

Yes.

[Felix Salmon]

And the model here is that if you cross sell more Products to more people, you all make more money. And if you do that in a kind of. If you just look at a bank which doesn't have these quotas, and you will obviously see that the customers which have the most products are the most profitable customers for the bank. But then the minute you start putting in the quotas, you game the system, essentially, and you wind up giving a whole bunch of products to people who don't want them and who are not going to be more profitable. So I.

[Cathy O'Neil]

You know, basically, what you're saying is weapons of mass destruction is a new lens through which you see absolutely everything.

[Jordan Weissman]

We are the only ones.

[Cathy O'Neil]

I have.

[Jordan Weissman]

A lot of people brought this up. You get more of what you measure. That's exactly.

[Felix Salmon]

Even if it's fake accounts, My number is $182 million.

[Jordan Weissman]

Okay.

[Felix Salmon]

Which is. And this is. I can't believe we managed to go this far in this. In this episode without using the word Facebook.

[Jordan Weissman]

But.

[Felix Salmon]

But Facebook is, of course, the world's biggest algorithm. Basically, it's just. It's a $100 billion algorithm, and everything you see is filtered through a million algorithms. And the most profitable one is the algorithm which shows you ads. But there are many, many others. $182 million is the amount of ad revenue that Facebook has. Just in Norway, which is a very small country, and right now, there is a huge fight in Norway because a Norwegian photographer put Nick's famous photograph of the girl being napalmed in Vietnam up on his. On a Facebook post. Facebook down. The photographer complained. Facebook wound up suspending him. The biggest newspaper in Norway came out and said, this is ridiculous. Published the photograph on their Facebook page. They got that post taken down. The prime minister of Norway came out and put the photograph on her Facebook page, and she had that post taken down. And there was this massive fight.

[Cathy O'Neil]

This is like a week after Zuckerberg said that they're not a media company.

[Felix Salmon]

Yeah. And so there's this massive fight where Facebook is going after the biggest publisher in Norway, the prime minister of Norway, and basically telling them what they can and can't publish. Mark Zuckerberg is saying that he's not a publisher, and Facebook is making $182 million a year out of Norway. Virtually none of which gets sort of filtered back into Norway itself. It just gets dividended back to Silicon Valley. So that is the fait du jour, which I am now, as you say, Kathy, looking at through the lens of your yellow book, is there any way.

[Jordan Weissman]

That Norway wins this fight? Like, is there any Context in which.

[Cathy O'Neil]

They start their own Facebook.

[Felix Salmon]

Yeah, that's not gonna happen.

[Jordan Weissman]

Yeah, exactly.

[Cathy O'Neil]

My number is 16. I've been looking at the Airbnb discrimination stuff going on there.

[Felix Salmon]

They recently came out saying they're gonna try harder to avoid discrimination.

[Cathy O'Neil]

Yeah. So I think it's, you know, black sounding names are 16% more likely to be rejected by Airbnb hosts than white sounding names. So there's pretty good evidence that there's real discrimination on the part of the hosts who don't want black people staying at their house. And they have all sorts of design ideas for making that much, much harder, which I really like.

[Jordan Weissman]

Yeah, it is such a tricky question just because also you're dealing with not just a national company, but you're doing an international company that just. So many different culture mores.

[Cathy O'Neil]

You can fix a lot of things with just one design change. And the design change that I think is most important is like, basically an automatic acceptance. You don't get to decide. You're like, you either get to offer your room that night or you don't.

[Felix Salmon]

I'm old enough to remember when Airbnb was this thing where, like, it was all plugged into the Facebook graph and you would be like, I will only rent to you if your my friend or a friend of my friend.

[Cathy O'Neil]

Those are old days.

[Felix Salmon]

So, okay, that is it for the episode. I will reiterate my request. Oh, my other number, which I have to mention is 99, because Kathy's book was at 1 point number 99 on the overall Amazon bestseller list, which is kind of amazing.

[Jordan Weissman]

Did it really? I didn't know that.

[Felix Salmon]

Let's get it higher than that.

[Jordan Weissman]

Push it up, push it up.

[Felix Salmon]

Buy it in your store. Okay, yeah, yeah, on Amazon or anywhere on Audible or anywhere else. And write to us. The address is slatemoneylate.com if you read the book, if you have any feedback for Kathy, she will get it that way. I do need to thank Veralyn Williams, the producer of this show, and the executive producers, Steve Lichti and Andy Bowers and the whole Panoply network, which you can [e-mail] panoply so we will talk to you next week on Sleep Money.