Biased Algorithms, Biased World

Summary

Podcast Summary: On the Media – "Biased Algorithms, Biased World"

Introduction

In the September 1, 2021, episode of On the Media titled "Biased Algorithms, Biased World," hosts Brooke Gladstone and Micah Loewinger delve into the pervasive influence of algorithms in modern society. They explore how these mathematical constructs, often perceived as neutral tools, can perpetuate and exacerbate social inequalities. Central to the discussion is Cathy O'Neill, mathematician, data scientist, and author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.

Meet Cathy O'Neill

Cathy O'Neill brings a wealth of expertise to the conversation. As the founder of the consulting firm ORCAA, which audits algorithms for biases related to race, gender, and economic status, she provides critical insights into the hidden dangers of unchecked algorithmic applications. O'Neill's background as a former Wall Street quant and her pivot to investigating the societal impacts of algorithms following the 2008 financial meltdown underscore her commitment to uncovering the flaws in data-driven decision-making.

Understanding Algorithms

Brooke Gladstone initiates the discussion by asking O'Neill to define what an algorithm is. O'Neill clarifies:

“It's just a set of directions. Long division that you learn in fourth grade is an algorithm. I use the word algorithm. It's short for predictive algorithm, and that's a way of predicting the future based on the past.” (02:18)

She emphasizes that algorithms rely heavily on "training data," which comprises historical information used to forecast outcomes such as loan approvals, hiring decisions, and even criminal sentencing.

Weapons of Math Destruction (WMDs)

O'Neill introduces the concept of Weapons of Math Destruction (WMDs), distinguishing them from benign algorithms. She outlines three defining characteristics of WMDs:

1. Opacity: Their inner workings are often hidden from public view.
2. Scale: They affect a large number of people.
3. Damage: They have destructive impacts, particularly on marginalized groups.

“Most algorithms are totally benign... Algorithms that are important, that are destructive and that are secret. That's the weapon of mass destruction.” (02:54)

Examples of WMDs

1. Predictive Policing:

   O'Neill discusses how predictive policing algorithms use arrest data as a proxy for actual criminal activity. This approach is flawed because it disproportionately targets neighborhoods with higher police presence, often overburdening communities of color.

   “When we use arrests as a proxy for crime, we are really overburdening those people who are already profiled by the police.” (04:19)

2. Hiring Algorithms:

   In the realm of employment, algorithms assess candidates based on historical data that may embed biases related to promotions, raises, and employee retention.

   “Implicit bias that we know exists... gets baked into the algorithm that I just wrote.” (07:40)

3. Facial Recognition Technology:

   O'Neill highlights the technical shortcomings of facial recognition systems, particularly their lower accuracy rates for women and people of color due to biased training datasets.

   “They weren't thinking carefully enough before deploying it to the world to say, hey, does this work? As well on black faces as white faces.” (09:02)

Impact on Society

The discussion underscores how biased algorithms can entrench systemic inequalities. By relying on flawed proxies and opaque processes, these algorithms often disadvantage already marginalized groups, reinforcing existing societal disparities.

Case Studies

1. Clopenings in Labor Scheduling:

   O'Neill introduces the concept of "clopenings," where workers experience unpredictable schedules that prevent them from qualifying for benefits or maintaining a stable work-life balance. She describes how scheduling algorithms can be manipulated to minimize benefits eligibility at the expense of employee well-being.

   “It's a small benefit for the employer... To make sure that none of your employees get enough hours a week to qualify for benefits... Their life [is wrecked].” (11:13)

2. Starbucks Scheduling Algorithm:

   Despite promises to eliminate clopenings, Starbucks' scheduling practices continue to prioritize efficiency over fairness, demonstrating the difficulties in altering algorithm-driven systems even when ethical concerns are raised.

   “The incentives to managers to be efficient were so irresistible that they never actually made any changes.” (12:54)

3. U.S. News & World Report College Rankings:

   O'Neill critiques how college ranking algorithms incentivize institutions to "game" the system, focusing on metrics that may not align with educational quality or affordability. This creates a feedback loop where colleges adjust their practices to meet ranking criteria rather than actual educational needs.

   “Colleges know that if they look exclusive, then they look better for the ranking. They just get a bunch of kids they know will never make it to apply.” (14:28)

The Problem with Training Data and Proxies

A recurring theme in the conversation is the reliance on historical data and proxies that may inherently contain biases. Whether predicting criminal behavior based on arrest records or hiring based on past employee success metrics, the choice of data inputs can perpetuate existing inequalities.

“Algorithms that predict behavior... Not on what you do or what you've done... but who you are.” (08:25)

Lack of Standards and Accountability

O'Neill emphasizes the absence of standardized regulations governing algorithmic transparency and fairness. Proprietary algorithms remain black boxes, leaving users and those affected by them without recourse or understanding.

“Nobody... there is no standard. A large company says, oh, we don't want to build these algorithms, we want to rent them... We don't have to explain it at all.” (10:25)

Efficiency vs. Fairness

The episode raises a critical philosophical question: Should society prioritize efficiency over fairness? O'Neill argues that current algorithmic applications are skewed towards profitability, often at the expense of human well-being and equity.

“Do we have any answer to capitalism?... Algorithms... about profitability, not about happiness.” (13:16)

Public Perception and Control

O'Neill advocates for increased public skepticism and agency regarding algorithmic decisions. She urges individuals to question and understand the algorithms that impact their lives, highlighting that societal control over these tools is paramount.

“I want us to learn to be skeptical... The power that we give to the algorithms is the thing we have the most control over.” (13:40)

Key Takeaways

1. Algorithms Are Not Neutral: They reflect the biases present in their training data and the intentions of their creators.
2. WMDs Pose Significant Risks: Opacity, scale, and destructive impacts make certain algorithms particularly harmful.
3. Need for Transparency and Regulation: Establishing standards for algorithmic fairness and accountability is crucial.
4. Redefining Success: Society must critically assess and redefine what constitutes success, ensuring it aligns with human values rather than corporate profitability.
5. Empowerment Through Skepticism: Encouraging public awareness and questioning of algorithmic decisions can help mitigate their negative impacts.

Conclusion

The "Biased Algorithms, Biased World" episode of On the Media provides a compelling examination of how algorithms, while seemingly impartial, can perpetuate and even amplify social injustices. Cathy O'Neill's insights urge listeners to critically evaluate the role of data-driven decision-making in society and advocate for greater transparency and fairness in algorithmic applications.

Transcript

[Unknown Announcer]

On the Media is brought to you by Progressive Insurance. Do you ever think about switching insurance companies to see if you could save some cash? Progressive makes it easy to see if you could save when you bundle your home and auto policies. Try [e-mail] Progressive Casualty Insurance Company and affiliates. Potential savings will vary. Not available in all states.

[Brooke Gladstone]

This is on the Media. I'm Brooke Gladstone. So while Congress sweats it out during their summer recess, we decided to take a break from politics, take refuge in the uncontroversial world of numbers. After all, addition is neutral, right? For much of the 21st century, it seemed that if you had a problem, you could just, you know, throw an algorithm at it. Algorithms are replacing human advisors and brokers.

[Cathy O'Neill]

Our military uses an algorithm in their.

[Brooke Gladstone]

Skynet program to decide who should be.

[Cathy O'Neill]

On the terrorism kill list.

[Unknown Announcer]

AI is being used for everything from diagnosing illnesses to helping police predict crime hotspots.

[Brooke Gladstone]

The only problem? A lot of times, they just don't work. Stanley Tucci may have said it best in the midst of a robotic production snafu in the movie Age of Extinction.

[Cathy O'Neill]

Algorithms.

[Brooke Gladstone]

Math. Why can't we make what we want to make the way we want to make it? Why? Yeah, why? Well, in 2019, we sat down with slightly less dramatic flair to ask Kathy o' Neill, mathematician, data scientist, and investigative journalist, that very question. She founded the consulting firm orcaa. Orca? Which audits algorithms for racial, gender, and economic inequality and all around bad science. She loves math. She used to be a Wall street quant, but something about the financial meltdown of 2008 turned her off the use of algorithms for the purposes of prediction. Something about how no one actually checks to see if they really work and what happens when they don't. And even when they do. She's the author of Weapons of Math. How Big data Increases inequality and Threatens democracy. I started with the basics. What's an algorithm?

[Cathy O'Neill]

It's just a set of directions. Long division that you learn in fourth grade is an algorithm. I use the word algorithm. It's short for predictive algorithm, and that's a way of predicting the future based on the past. And we use the training data that is all around us.

[Brooke Gladstone]

When you say training data.

[Cathy O'Neill]

The information we've collected from the past, like it's memories, they are used to.

[Brooke Gladstone]

Select to whom to give or deny a loan.

[Cathy O'Neill]

That's right.

[Brooke Gladstone]

Who gets hired and who doesn't.

[Cathy O'Neill]

Who goes to prison and for how long.

[Brooke Gladstone]

There are algorithms and there are weapons of math destruction. What's the difference?

[Cathy O'Neill]

Most algorithms Are totally benign. Like I could build an algorithm in my basement on my computer. I could be trying to predict the stock market and nobody uses it. It doesn't matter. So I guess the shorthand version is algorithms that are important, that are destructive and that are secret. That's the weapon of mass destruction.

[Brooke Gladstone]

Let's talk about some algorithms that you note in your book that are maybe one of those things, but not the others. For instance, sports algorithms predicting how teams or players may behave fed data that actually reflect the behavior that they're trying to predict. They are regularly updated, and though they are widely used, you wouldn't regard them as a weapon of math destruction.

[Cathy O'Neill]

Correct. So they're very important, like in the sense that there's a lot of money behind them and people really care if they're right or wrong. But if they make a mistake, that gets learned. So if we don't trade for a player and they go to another team and do really well, the algorithm learns that they made a mistake. And that's often not the case for weapons of mass destruction. But the real thing that distinguishes that is that it doesn't wreak havoc.

[Brooke Gladstone]

Can you give me an example of using proxies? You can't actually use the real thing like an athlete's behavior in the league. You have to use something that might be an indicator of something else. That's a proxy that characterizes a lot of your wmd.

[Cathy O'Neill]

Absolutely.

[Brooke Gladstone]

Give me an example of that.

[Cathy O'Neill]

Well, the most pernicious example of that is, in my opinion, the predictive policing or the crime risk scores, the recidivism risk scores.

[Brooke Gladstone]

Police in Los Angeles are trying to predict the future.

[Cathy O'Neill]

We know where crime happened yesterday, but where is it going to happen tomorrow and the next day?

[Brooke Gladstone]

And they're not alone. More and more departments are using data driven algorithms to forecast crime.

[Cathy O'Neill]

So they predict locations of arrests to say that's where the crime must be, Rather than acknowledging that police act differently in certain neighborhoods than they act in other neighborhoods. We don't really have crime data. If you think about it. There's lots of crimes that go on that do not lead to arrests. There's lots of pot smoking among white people that never get arrested. So there's a lot more sort of non arrested white crime than people of color. So when we use arrests as a proxy for crime, we are really overburdening those people who are already profiled by the police.

[Brooke Gladstone]

So in that case, arrests are used instead of criminal behavior.

[Cathy O'Neill]

Yes. When I say something like arrests are a bad proxy for crime, I'M sure a lot of your listeners are like, but people get arrested for crimes. You know, I just want to make a point that, you know, I've talked to a lot of police chiefs and a lot of judges about these kinds of algorithms, and one of the things they keep on coming back to is almost no real mental health care in this country. So people get arrested very consistently for addiction problems or untreated mental health problems. That's not crime. The police don't think it's crime. The judges don't really want to think of it as crime. And yet these scoring systems are basically suggesting, since this person is much more likely to be rearrested in the future, because guess what? They're still gonna be addicted, or they're still gonna have a mental health problem, or they're still gonna be poor. And there's all sorts of crime, criminal crimes.

[Brooke Gladstone]

They're still gonna be living in a neighborhood where behavior that could go on pretty much unchallenged in a frat house is gonna send you into the system.

[Cathy O'Neill]

Exactly. So those very predictable things show up quite well statistically. So that is how your score goes up. That's how you get to that point where you're considered high risk, and you're actually sentenced to prison for longer. And the judges, don. Judges want to be putting people into prison because they're actually a public health risk, not because they're addicts.

[Brooke Gladstone]

That's a proxy that characterizes a lot of your wmd.

[Cathy O'Neill]

Everybody will understand hiring algorithms. So let's just say you have a big company, Brooke. You're just like, oh, my God, I'm getting so many applications for these 10 positions. And, like, I get a thousand applications. How am I going to sort through them all? I would like somebody to help me. And I don't want it to be a person because they're too expensive. I want it to be an algorithm. And. And so you hire me. I'm your data scientist, and I come and build you an algorithm to sort through these applications. Well, you're going to say, kathy, I want to hire people that will be successful at my firm. And I'll say, okay, well, what do you mean by success? This is where the proxy comes in. You'll say, well, I don't measure directly whether someone's good at their job, because you can't know that, because how do you know that? Right? I mean, what does it mean for someone to do well at your company? What do you think?

[Brooke Gladstone]

Okay, they stay.

[Cathy O'Neill]

They stay a long time.

[Brooke Gladstone]

They generate a lot of good Ideas.

[Cathy O'Neill]

Oh, how do you measure that, though?

[Brooke Gladstone]

Let's see. They get promoted.

[Cathy O'Neill]

Excellent. Okay, so we have this kind of triumvirate of data points about each employee, like how long do they stay, how many promotions, how many raises. This is exactly the way that people measure success at companies. And this is exactly the kind of algorithm that gets built. So I'm training your data on 20 years of your past practice of hiring people. Boom. Implicit bias that we know exists in who gets promoted, who gets hired, who gets raises, who stays for a long time, who feels welcome enough to stay for a long time, gets baked into the algorithm that I just wrote.

[Brooke Gladstone]

And you observe in the book that almost all of these algorithms predict behavior. Not on what you do or what you've done, because that's so hard to measure, but who you are.

[Cathy O'Neill]

And I'll just add one last point to emphasize how invisible this might look from the perspective of the employer. Certain kinds of mistakes are much more obvious when you do it this way, namely, false positives, which is to say, I've hired someone, they didn't work out. That is easy to spot, because what a pain. What you don't see are the people that you could have hired that would have worked out, but were filtered out. And that's where we see the sort of narrowing bottleneck of who is deemed future successful.

[Brooke Gladstone]

And another thing we don't really understand how it works, but we have to worry about its implications is facial recognition technology.

[Cathy O'Neill]

Yes.

[Brooke Gladstone]

What's the problem there?

[Cathy O'Neill]

Well, there's a bunch of different levels of problems. One of them is that it sometimes just doesn't work. So my friend Joy Bilamwini at MIT Media Lab was the first to come out with an audit. She's done a few audits now, Amazon most recently, where she found that at a technical level, they weren't working very well.

[Brooke Gladstone]

All companies perform better on males than females, and all companies also performed better on lighter subjects than on darker subjects. We saw that all companies performed worst on darker females. In fact, as we tested women with darker and darker skin, the chances of being correctly gendered came close to a coin toss. So why. Why does facial recognition work better on white men than black women?

[Cathy O'Neill]

It doesn't have to. It just happens to. Because of the training data. Literally the corpus of pictures that were used to train the algorithm was much more white and much more male. I think Joy calls them the pale male data problem. And believe it or not, they weren't thinking carefully enough before deploying it to the world to say, hey, does this work? As well on black faces as white faces. Why don't these companies get ahead of this a little bit and test this and sort of have evidence in advance that this is not going to be unfair?

[Brooke Gladstone]

Who determines if the algorithms are working and how?

[Cathy O'Neill]

That's kind of the craziest part, and I'm so glad you asked. Nobody. There is no standard. A large company says, oh, we don't want to build these algorithms, we want to rent them. Essentially we license them from some data vendor. And the data vendor says, oh, you can trust this, but we're not going to explain it at all. It's a black box, proprietary. That's part of the licensing agreement. You don't get to know how this works, but you can use it to hire people. You can use it if you're a police department to find people. You can use it if your department of education to fire your teachers, blah, blah, blah. There is no particular standard.

[Brooke Gladstone]

Which brings us to the issue of how do you determine when an algorithm is successful? What is your definition of success? I was really moved by your discussion of clopenings.

[Cathy O'Neill]

Yeah, this is a great example of where the definition of success for the people using the algorithm is the opposite as a definition of success for the people who are targeted by the algorithm. So clopenings is the concept where you are basically a minimum wage worker, probably working in a large store, and you close on one evening and then you open the store the next morning and you probably don't have enough time to even go home and see your kids. Right? Barely enough time to sleep. And the crazy thing is that these scheduling algorithms will, for one week make you clope in three days in a row, and then the next week you don't have any work at all. I looked into the research that was developing these algorithms. One of them made me cry. I mean, it was so brutal. It was like you have the option, if you use this algorithm, to toggle this switch to make sure that none of your employees get enough hours a week to qualify for benefits. You can just turn this little switch on and like all of your employees will be wage slaves forever. They will not be able to go to night school because their hours change every day. They will be able to put their kids in daycare regularly. It is such a small benefit for the employer. If you compare it to the wrecking of the life of the employee. It's maddening, but it's not actually technically illegal. So the algorithm exploits it.

[Brooke Gladstone]

You pose the question, should we as a society be willing to sacrifice a little Efficiency in the interest of fairness. And you talk about Starbucks. Starbucks wants to have a good image. Its scheduling algorithm was exposed.

[Cathy O'Neill]

Yeah.

[Brooke Gladstone]

It said that it was going to improve it. No more clopenings, no more employing people, short of triggering some benefits. Right, but the trouble was that the incentives to managers to be efficient were so irresistible that they never actually made any changes.

[Cathy O'Neill]

Yeah, I mean, it's a philosophical question. It's basically you're saying, do we have any answer to capitalism? You know, all these algorithms that they're using in these corporate settings are about profitability, not about happiness. So if we wanted to address that, we would actually have to change the incentive structure of corporations. It's a big ask.

[Brooke Gladstone]

How would you assess the way that we, the general public, view algorithms?

[Cathy O'Neill]

I want us to learn to be skeptical. I want us to say, I don't need a math PhD to ask you why I'm getting fired. The power that we give to the algorithms is the thing we have the most control over.

[Brooke Gladstone]

Do we?

[Cathy O'Neill]

Let me give you another example. The U.S. news & World Report college ranking model. Who gives that power? Us.

[Brooke Gladstone]

You describe the impact of that ranking. Colleges turn themselves into pretzels. Students spend tons of money in order to fit the parameters that colleges have adapted to because of the rankings in U.S. news and world Report. A pernicious feedback loop, you call it.

[Cathy O'Neill]

They're bogus and they're gameable. A college knows that if they look exclusive, then they look better for the ranking. So they just get a bunch of kids they know will never make it to apply. And yet, of all the stupid things that the US News and World Report pays attention to, it doesn't pay attention to the cost.

[Brooke Gladstone]

It's not one of the criteria.

[Cathy O'Neill]

Exactly. When college admissions officers are crazily gaming the algorithm, which is what their job seems to be nowadays, they don't care if the tuition goes up. Why do we keep giving these questionable, stupid algorithms so much power?

[Brooke Gladstone]

If people listening to this interview only take one thing away, this is what I'd want them to take away. You say that an algorithm is an opinion embedded in math, Right?

[Cathy O'Neill]

I mean, there's so many choices that go into every algorithm, and the most important one being, what do we mean by success? If I get to define success for myself, that's one thing, but if Facebook is defining success for me, I don't trust it. As algorithms proliferate, which they are applying for credit, applying for a job, applying to go to college, applying for a loan, applying for housing, all those things are now algorithmic. So they all define success for them, not for us. That's their opinion. Right? It's really, really important to remember that it's not necessarily opinion that you have to share.

[Brooke Gladstone]

Thank you so much.

[Cathy O'Neill]

Thank you so much, Brooke.

[Brooke Gladstone]

We spoke with Cathy O' Neill in 2019. She is a mathematician, data scientist, founder of the consulting firm orca, and the author of Weapons of Math How Big Data Increase Inequality and threaten democracy.

[Unknown Announcer]

Since WNYC's first broadcast in 1924, we've been dedicated to creating the kind of content we know the world needs. In addition to this award winning reporting, your sponsorship also supports inspiring storytelling and extraordinary music that is free and accessible to all. To get in touch and find out more, visit sponsorship.wnyc.org.