← Radiolab
Radiolab
Driverless Dilemma
00:40:06
Most of us would sacrifice one person to save five. It’s a pretty straightforward bit of moral math. But if we have to actually kill that person ourselves, the math gets fuzzy. That’s the lesson of the classic Trolley Problem, a moral puzzle that fried our brains in an episode we did about 11 years ago. Luckily, the Trolley Problem has always been little more than a thought experiment, mostly confined to conversations at a certain kind of cocktail party. That is until now. New technologies are forcing that moral quandry out of our philosophy departments and onto our streets. So today we revisit the Trolley Problem and wonder how a two-ton hunk of speeding metal will make moral calculations about life and death that we can’t even figure out ourselves. This story was reported and produced by Amanda Aronczyk and Bethel Habte. Thanks to Iyad Rahwan, Edmond Awad and Sydney Levine from the Moral Machine group at MIT. Also thanks to Fiery Cushman, Matthew DeBord, Sertac Karaman, Martine P...
Loading summary
Transcript291 lines
- [00:00]
A
With Venmo Stash. A taco on one hand and ordering a ride in the other means you're stacking cash back with Venmo Stash. Get up to 5% cash back when you pick a bundle of your favorite brands. Earn more cash when you do more with Stash. Venmo Stash terms and exclusions apply. Match $100 cash back per month. See terms of Venmo Me Stashterms. For delicious meals, you could go out.
- [00:17]
B
To eat, or you could just make a Marie Callender's meal.
- [00:20]
A
Marie Callender's classic chicken Parmigiano bowl is so good.
- [00:24]
B
It has marinara sauce that's made from scratch and creamy mozzarella cheese over pasta.
- [00:29]
A
It's delicious with no artificial flavors, colors or preservatives.
- [00:33]
B
And 30 grams of protein. You can find it in the frozen aisle. Marie, what having it all tastes like.
- [00:41]
A
It's never too early for Lowe's Black Friday deals. Snag some of our biggest savings of the season right now, like 25% off.
- [00:48]
B
Select pre lit artificial Christmas trees. And get yourself free select Dewalt cobalt or Craftsman tools when you buy a select battery or combo kit before the Black Friday rush.
- [00:59]
A
Because everyone loves free stuff, right?
- [01:01]
C
Lowe's.
- [01:02]
B
We help you save. Valid through 123 while supplies last. Selection varies by location.
- [01:11]
C
Wait, you're listening?
- [01:12]
A
Okay. All right. Okay. All right.
- [01:18]
B
You're listening to Radiolab Radio from wnyc. I'm Jad Abumrad.
- [01:30]
A
I'm Robert Krulwich. And you know what this is?
- [01:32]
B
This is Radiolab.
- [01:33]
A
Yeah.
- [01:33]
B
Okay, so we're gonna play you a little bit of tape first just to set up what we're gonna do today. About a month ago, we were doing the thing about the fake news.
- [01:40]
A
Yeah, we were very worried about a lot of fake news. A lot of people are. But in the middle of doing that reporting, we were talking with a fellow from Vanity Fair.
- [01:47]
D
My name is Nick Bilton. I'm a special correspondent for Vanity Fair.
- [01:50]
A
And in the course of our conversation, Nick. And this had nothing to do with what we were talking about, by the way. Nick just got into a sort of a. Well, he went into a kind of a nervous reverie, I'd say.
- [02:01]
B
Yeah, he was like, you know, you guys want to talk about fake news, but that's not actually what's eating at me.
- [02:05]
D
The thing that I've been pretty obsessed with lately is actually not fake news. But it's automation and artificial intelligence and driverless cars because it's going to have a Larger effect on society than any technology that I think has ever been created in the history of mankind. I know that's kind of a bold statement, but quite bold. But you've got to imagine that there will be in the next 10 years, 20 to 50 million jobs that will just vanish to automation. You've got million truckers that will lose their jobs. But we think about automation and driverless cars and we think about the fact that they are going to the people that just drive the cars, like the taxi drivers and the truckers are going to lose their jobs. What we don't realize is that there are entire industries that are built around just cars. So for example, if you're not driving the car, why do you need insurance? There's no parking tickets because your driverless car knows where it can and cannot park and goes and finds a spot and moves and so on. If there are truckers that are no longer using rest stops because driverless cars don't have to stop and pee or take a nap, then all of those little rest stops all across America are affected. People aren't stopping to use the restrooms, they're not buying burgers, they're not staying in these hotels, and so on and so forth. And then you look at driverless cars to a next level. The whole concept of what a car is is going to change. So, for example, right now a car has five seats and a wheel. But if I'm not driving, well, what's the point of having five seats and a wheel? You could imagine that you take different cars. And maybe when I was on my way here to this interview, I wanted to work out. So I called a driverless gym car or I have a meeting out in Santa Monica after this and it's an hour. So I call a movie car to watch a movie on the way out there, or an office car and I pick up someone else and we have a meeting on the. All of these things are gonna happen, not in a vacuum, but simultaneously. This, you know, pizza delivery drivers are gonna be replaced by robots that will actually cook your pizza on the way to your house in a little box and then deliver it. And so, kind of a little bit of a long winded answer, but I truly do think that it's gonna have a massive, massive effect on society. Am I stressing you guys out? Are you having heart palpitations over there?
- [04:22]
A
Yes, my blood pressure has gone down. So that's a fairly compelling description of a very dangerous future.
- [04:30]
B
Yes. You know what, it's funny. One of the things that I Mean, we couldn't use that tape initially, at least, but we kept thinking about it because it actually weirdly points us back to a story we did about a decade ago, the story of a moral problem that's about to get totally reimagined.
- [04:48]
A
It may be that what Nick is worried about and what we were worried about 10 years ago have now come dangerously close together.
- [04:55]
B
So what we thought we would do is we're gonna play you the story as we did it, then sort of the full segment, and then we're going to amend it on the back end. And by way of just disclaiming, this was at a moment in our development where there's just, like, way too many sound effects. Just gratuitous.
- [05:11]
A
Well, you don't have to apologize for this.
- [05:12]
B
No, I'm going to apologize because it was just too much. Just too much. And also, like, we. We talk about the FMRI machine. Like, it's this, like, amazing thing.
- [05:21]
D
When was.
- [05:22]
B
It's sort of commonplace now. Anyhow, doesn't matter. We're going to play it for you and then talk about it on the back end. This is. We start with. With a description of something called the trolley problem. You ready?
- [05:35]
A
Yep.
- [05:36]
B
All right. You're near some train tracks. Go there in your mind.
- [05:40]
A
Okay.
- [05:41]
B
There are five workers on the tracks working. They've got their backs turned to the trolley, which is coming in the distance.
- [05:48]
A
You mean they're repairing the tracks?
- [05:49]
B
They are repairing the tracks.
- [05:50]
A
This is unbeknownst to them, the trolley is approaching.
- [05:52]
B
They don't see it. You can't shout to them.
- [05:55]
A
Okay.
- [05:55]
B
And if you do nothing, here's what'll happen. HIVE workers will die.
- [06:04]
A
Oh, my God. That was a horrible experience. I don't want that to happen.
- [06:08]
B
No, you don't. But you have a choice. You can do A, nothing, or B, it so happens next to you is a lever. Pull the lever, and the trolley will jump onto some side tracks where there is only one person working.
- [06:22]
A
So if the song. So if the trolley goes on the second track, it will kill the one guy.
- [06:28]
B
Yeah. So there's your choice. Do you kill one man by pulling a lever, or do you kill five men by doing nothing?
- [06:35]
A
Well, I'm gonna pull the lever, naturally.
- [06:39]
B
All right, here's part two. You're standing near some train tracks. Five guys are in the tracks, just as before. And there is the trolley.
- [06:47]
A
The same five guys working, five guys backs to the train. They can't see anything. Yeah.
- [06:51]
B
Yeah, exactly. However, I'm making a couple changes. Now. You're Standing on a footbridge that passes over the tracks. You're looking down onto the tracks. There's no lever anywhere to be seen except next to you. There is a guy.
- [07:03]
A
What do you mean, there's a guy?
- [07:04]
B
A large guy, large individual standing next to you on the bridge, looking down with you over the tracks, and you realize, wait, I can save those five workers. If I push this man, give him.
- [07:13]
C
A little tap.
- [07:18]
B
He'll land on the tracks and he stops the train, right?
- [07:22]
A
Oh, yeah. I'm not gonna do that. I'm not gonna do that.
- [07:25]
B
But surely you realize the math is the same.
- [07:28]
A
You mean I'll save four people this way?
- [07:30]
B
Yeah.
- [07:30]
A
Yeah. But this time I'm pushing the guy. Are you insane? No.
- [07:34]
B
All right, here's the thing. If you ask people these questions, and we did, starting with the first, is it okay to kill one man to save five using a lever? 9 out of 10 people will say, yes, yes, yes. Yeah. But if you ask them, is it okay to kill one man to save five by pushing the guy? Nine out of 10 people will say, no, no, never.
- [07:54]
A
No, no.
- [07:55]
B
It is practically universal. And the thing is, if you ask people, why is it okay to murder. That's what it is. Murder a man with a lever and not okay to do it with your hands? People don't really know pulling the lever to save the five. No, that feels better than pushing the one to save the five. But I don't really know why. So that's a good. There's a good moral quandary for you.
- [08:26]
A
And if having a moral sense is a unique and special human quality, then maybe we. We us two humans. Anyway, you and me, should at least inquire as to why this happens. And I happen to have met somebody who has a hunch. He's a young guy at Princeton University. Wild, curly hair, bit of mischief in his eye. His name is Josh Green, and he spent the last few years trying to figure out where this inconsistency comes from.
- [08:56]
C
How do people make this judgment? Forget whether or not these judgments are right or wrong. Just what's going on in the brain that makes people distinguish so naturally and intuitively between these two cases, which, from an actuarial point of view, are very, very, very similar, if not identical.
- [09:13]
A
Josh is, by the way, a philosopher and a neuroscientist. So this gives him special powers. He doesn't sort of sit back in a chair, smoke a pipe and think, now, why do you have these differences? He says, no, I would like to look inside people's heads. Because in our heads, we may find clues as to where these feelings of revulsion or acceptance come from in our brains.
- [09:36]
C
So we're here in the control room. What you basically just see is.
- [09:39]
A
And it just so happens that in the basement of Princeton, there was this. Yeah, yeah, well, big circular thing.
- [09:46]
C
Yeah, it looks kind of like an airplane engine.
- [09:48]
A
180,000 pound brain scanner.
- [09:52]
C
I'll tell you a funny story. You can't have any metal in there because the magnet. So we have this long list of questions that we ask people to make sure they can go in. Do you have a pacemaker? Have you ever worked with metal? Blah, blah, blah, blah.
- [10:04]
A
Have you ever worked with metal?
- [10:05]
C
Yeah, because you can have little flecks of metal in your eyes that you would never even know are there from having done metalworking. And one of the questions is whether or not you wear a wig or anything like that, because they often have metal wires and like that. And there's this very nice woman who does brain research here, who's Italian, and she's asking her subjects over the phone all these screening questions. And so I have this person over to dinner. She says, yeah, you know, I ended up doing this study, but it has you the weirdest questions. This woman's like, do you have a herpes? And I'm like, what does it have to do if I have herpes or not? If I want to, anyway. And she said, you know, she asked me, she said, do you have a hairpiece? But she. So now she asks people if you wear a wig or whatever.
- [10:44]
A
Anyhow, what Josh does is he invites people into this room, has them lie down on what is essentially a cot on rollers, and he rolls them into the machine. Their heads are braced so they're sort of stuck in there. Have you ever done this?
- [10:57]
C
Oh, yeah. Yep, several times.
- [10:59]
A
And then he tells them stories. He tells them the same two, you know, trolley tales that you told before. And then at the very instant that they're deciding whether I should push the lever or whether I should push the man. At that instant, the scanner snaps pictures of their brains. And what he found in those pictures was, frankly, a little startling. He showed us some.
- [11:20]
C
I'll show you some stuff. Okay, let me think.
- [11:25]
A
The picture that I'm looking at is a sort of a. It's a brain looked, I guess, from the top down.
- [11:30]
C
Yeah, it's top down and sort of sliced, you know, like a deli slicer.
- [11:34]
A
And the first slide that he showed me was a human brain being asked the question, would you pull the Lever. And the answer in most cases was yes. Yeah, I'd pull the lever when the brain's saying yes. You'd see little kind of peanut shaped.
- [11:46]
C
Spots of yellow, this little guy right here and these two guys right there.
- [11:50]
A
The brain was being active in these places. And oddly enough, whenever people said yes, yes, yes, yes to the lever question, the very same pattern lit up. Then he showed me another slide. This was a slide of a brain saying, no, no, I would not push the man. I will not push the large man. And in this picture, this one we're looking at here, this, it was a totally different constellation of regions that lit up. This is the no, no, no crowd.
- [12:16]
C
I think this is part of the no, no, no crowd.
- [12:18]
B
So when people answer yes to the lever question, there are places in their brain which glow.
- [12:24]
A
But when they answer, no, I will not push the man, then you get a completely different part of the brain.
- [12:28]
B
Lighting up, even though the questions are basically the same. What does that mean? What does Josh make of this?
- [12:35]
A
Well, he has a theory about this.
- [12:36]
C
A theory not proven, but I think this is what I think the evidence suggests.
- [12:40]
A
He suggests that the human brain doesn't hum along like one big unified system. Instead, he says, maybe in your brain, in every brain, you'll find little warring tribes, little subgroups, one that is sort of doing a logical sort of counting kind of thing.
- [12:57]
C
You've got one part of the brain that says, huh, five lives versus one life. Wouldn't it be better to say five versus one?
- [13:03]
A
And that's the part that would glow when you answer, yes, I'd pull the lever. Yeah, I pull the lever. But there's this other part of the brain which really, really doesn't like personally killing another human being and gets very upset, set at the fat man case and shouts, in effect, no, no.
- [13:18]
C
It understands it on that level and says, no, no, bad.
- [13:22]
B
Don't do no, I don't think I've come push.
- [13:25]
A
No, never.
- [13:26]
C
A person, instead of having sort of one system that just sort of churns out the answer and bing, we have multiple systems that give different answers and they duke it out. And hopefully out of that competition comes morality.
- [13:43]
A
This is not a trivial discovery that you struggle to find right and wrong depending upon what part of your brain is shouting the loudest. This is. It's like Bleacher's morality.
- [13:55]
B
Do you buy this?
- [13:56]
A
Hmm. You know, I just don't know. Yeah, I've always kind of suspected that a sense of right and wrong is mostly stuff that you get from your mom. And your dad. And from experience that it's culturally learned for the most part. Josh is kind of a radical in this respect. He thinks it's biological, I mean deeply biological, that somehow we inherit from the deep past a sense of right and wrong that's already in our brains from the get go.
- [14:25]
C
Before mom and dad, our primate ancestors, before we were full blown humans had intensely social lives. They have social mechanisms that prevent them from doing all the nasty things that they might otherwise be interested in doing. And so deep in our brain we have what you might call basic primate morality. And basic primate morality doesn't understand things like tax evasion, but it does understand things like pushing your buddy off of a cliff.
- [14:53]
A
So you're thinking then that the man on the bridge, that I'm on the bridge next to the large man and that I have hundreds of thousands of years of training in my brain that says don't murder the large man. Right. Whereas even if I'm thinking if I murder the large man, I'm going to save five lives and only kill the woman. But there's something deeper down that says don't murder the large man, right?
- [15:14]
C
Now that case, I think it's a pretty easy case, even though it's five versus one. In that case, people just go with what we might call the inner chimp. But there are other.
- [15:23]
A
The inner chimp is your unfortunate way of describing an act of deep goodness. Right.
- [15:29]
C
Well, that's what's interesting.
- [15:31]
A
The Ten Commandments for God, inner chimpanzees. Right.
- [15:33]
C
Well, what's interesting is that we think of basic human morality as being handed down from on high. And it's probably better to say that it was handed up from below. That our most basic core moral values are not the things that we humans have invented, but the things that we've actually inherited from other people. The stuff that we humans have invented are the things that seem more peripheral and variable.
- [15:56]
A
But something as basic as thou shalt not kill, which many people think was handed down in tablet form from a mountaintop from God directly to humans, no chimps involved.
- [16:08]
C
Right.
- [16:09]
A
You're suggesting that hundreds of thousands of years of on the ground training have gotten our brains to think, don't kill your kin, don't kill yourself.
- [16:19]
C
At least, you know, that should be your default response. I mean, certainly chimpanzees are extremely violent and they do kill each other, but they don't do it as a matter of course. They, so to speak, have to have some context sensitive reason for doing so.
- [16:33]
A
So now we're getting to the rub of it, you think that profound moral positions may be somehow embedded in brain chemistry.
- [16:43]
B
Yeah. And Josh thinks there are times when these different moral positions that we have embedded inside of us, in our brains, when they can come into conflict. And in the original episode, we went into one more story. This one you might call the Crying Baby Dilemma.
- [17:02]
C
The situation is somewhat similar to the last episode of MASH for people who are familiar with that. But the way we tell the story, it goes like this. It's wartime.
- [17:11]
B
There's an enemy patrol coming down the road.
- [17:13]
C
You are hiding in the basement with some of your fellow villagers.
- [17:16]
B
Let's kill those lights.
- [17:18]
C
And the enemy soldiers are outside. They have orders to kill anyone that they find.
- [17:23]
B
Quiet. Nobody make a sound until they've passed us.
- [17:27]
A
So there you are. You're huddled in the basement. All around you are enemy troops, and you're holding your baby in your arms, your baby with a cold, a bit of a sniffle. And you know that your baby could cough at any moment.
- [17:45]
C
If they hear your baby, they're gonna find you and the baby and everyone else, and they're gonna kill everybody. And the only way you can stop this from happening is cover the baby's mouth. But if you do that, the baby's going to smother and die. If you don't cover the baby's mouth, soldiers are gonna find everybody and everybody's gonna be killed, including you, including your baby.
- [18:03]
A
Then you have the choice. Would you smother your own baby to save the village, or would you let your baby cough, knowing the consequences?
- [18:16]
C
And this is a very tough question. People take a long time to think about it. And some people say yes, and some people say no.
- [18:23]
B
Children are a blessing and a gift.
- [18:26]
A
From God, and we do not do that to children.
- [18:28]
B
Yes, I think I would kill my baby to save everyone else and myself.
- [18:33]
A
No, I would not kill a baby.
- [18:34]
B
I feel because it's my baby, I have the right to terminate the life.
- [18:38]
A
I'd like to say that I would kill the baby, but I don't know if I'd have the inner strength. No. If it comes down to killing my own child, my own daughter, and my.
- [18:45]
C
Own son, then I choose death.
- [18:47]
A
Yeah, if you have to. Because it was done in World War II when the Germans were coming around. There was a mother that had a baby that was crying and rather to be found. She actually suffocated the baby, but the other people lived.
- [19:00]
B
Sounds like an old MASH thing. No, you do not kill your baby.
- [19:04]
A
In the final MASH episode, The Korean woman who's a character in this piece, she murders her baby. She killed it.
- [19:16]
B
She killed it. Oh, my God. Oh, my God. I didn't mean for her to kill it. I just wanted it to be quiet.
- [19:39]
A
It was a baby. She smothered her own baby. What Josh did is he asked people the question, would you murder your own child? While they were in the brain scanner, and at just the moment when they were trying to decide what they would do, he took pictures of their brains and what he saw. The contest we described before was global. In the brain, it was like a world war. That gang of accountants, that part of the brain was busy calculating, calculating. A whole village could die. A whole village could die. But the older and deeper reflex also was lit up, shouting, don't kill the baby. No, no, don't kill the baby. Inside, the brain was literally divided. Do the calculation. Don't kill the baby. Do the calculation. Two different tribes in the brain literally trying to shout each other out. And, Jed, this was a different kind of contest than the ones we talked about before. Remember before, when people were pushing a man off of a bridge, overwhelmingly, their brains yelled, no, no, don't push the man. And when people were pulling the lever, overwhelmingly. Yeah, yeah, pull the lever right there. It was distinct here. I don't think really anybody wins.
- [21:00]
B
Well, who breaks the tie? I mean, they had to answer something, right?
- [21:03]
A
That's a good question. And now is there a. Do you. What happens? Is it just two cries that fight each other out or is there a judge?
- [21:13]
C
Well, that's an interesting question. And that's one of the things that we're looking at.
- [21:17]
A
When you are in this moment with parts of your brain contesting, there are two brain regions.
- [21:23]
C
These two areas here towards the front.
- [21:25]
A
Right behind your eyebrows, left and right, that light up. And this is particular to us. He showed me a slide.
- [21:33]
C
It's those sort of areas that are very highly developed in humans as compared to other species.
- [21:38]
A
So when we have a problem that we need to deliberate over, the light, the. The front of the brain. This is above my eyebrow, sort of.
- [21:46]
C
Yeah, right about there.
- [21:48]
A
And there's two of them, one on the left and one on the right, bilateral. And they are the things that monkeys don't have as much of that we have.
- [21:56]
C
Certainly these parts of the brain are more highly developed in humans.
- [21:58]
A
Looking at these two flashes of light at the front of a human brain, you could say we are looking at what makes us special.
- [22:07]
C
That's a fair statement.
- [22:09]
A
A human being wrestling with a problem, that's what that is.
- [22:12]
C
Yeah, where it's both emotional, but there's also a sort of irrational attempt to sort of sort through those emotions. Those are the cases that are showing more activity in that area.
- [22:21]
B
So in those cases, when these dots above our eyebrows become active, what are they doing?
- [22:25]
A
Well, he doesn't know for sure, but what he found is in these close contests, whenever those nodes are very, very active, it appears that the calculating section of the brain gets a bit of a boost. And the visceral inner chimp section of the brain is kind of muffled. No, no. The people who chose to kill their children, who made what is essentially a logical decision over and over, those subjects had brighter glows in these two areas and longer glows in these two areas. So there is a definite association between these two dots above the eyebrow and. And the power of the logical brain over the inner chamber or the visceral brain.
- [23:11]
C
Well, you know, that's the hypothesis. It's going to take a lot of more research to sort of tease apart what these different parts of the brain are doing, or if some of these are just sort of activated in an incidental kind of way. I mean, we really don't know. This is all very new.
- [23:32]
B
Okay, so that was the story we put together many, many, many years ago, about a decade ago. And at that point, the whole idea of thinking of morality as kind of purely a brain thing, it's relatively new. And certainly the idea of philosophers working with FMRI machines, that was super new. But now here we are 10 years later and some updates. First of all, Josh Green.
- [23:52]
A
So in the. In the long, long stream of times, I assume now you have three giraffes, two bobcats and children.
- [24:00]
C
Yeah, so two kids. And we're close to adding a cat.
- [24:03]
B
We talked to him again. He has started a family. He's switched labs from Princeton to Harvard. But that whole time, that interim decade, he has still been thinking and working on the trolley problem.
- [24:14]
A
Did you ever write the story differently?
- [24:16]
C
Absolutely.
- [24:17]
B
For years, he's been trying out different permutations of the scenario on people. Like, okay, instead of pushing the guy off the bridge with your hands, what if you did it but not with your hands?
- [24:28]
C
So in one version, we asked people about hitting a switch that opens a trap door on the footbridge and drops the person. In one version of that, the switch is right next to the person. In another version, the switch is far away. And in yet another version, you're right next to the person and you don't push them off with your hands, but you Push them with a pole.
- [24:47]
B
And to cut to the chase, what Josh has found is that the basic results that we talked about, that's roughly held up still the case that people would like to save the most number of lives, but not if it means pushing somebody with their own hands or with a pole, for that matter. Now here's something kind of interesting. He and others have found that there are two groups that are more willing to push the guy off the bridge. They are Buddhist monks and psychopaths.
- [25:16]
C
I mean, some people just don't care very much about hurting other people. They don't have that kind of an emotional response.
- [25:21]
B
That would be the psychobats. Whereas the Buddhist monks presumably are really good at shushing their inner chimp, as he called it, and just saying to.
- [25:29]
C
Themselves, you know, I'm aware that this is. That killing somebody is a terrible thing to do, and I feel that. But I recognize that this is done for a noble reason and therefore it's okay.
- [25:41]
B
So there's all kinds of interesting things you can say about the trolley problem as a thought experiment, but at the end of the day, it's just that it's a thought experiment. What got us interested in revisiting it is that it seems like the thought experiment is about to get real. That's coming up right after the break. This is Amanda Darby calling from Rockville, Maryland.
- [26:19]
A
Radiolab is supported in part by the Alfred P. Sloan foundation, enhancing public understanding of science and technology in the modern world. More information about Sloan@www.sloan.org.
- [26:38]
B
Jad Robert, Radiolab. Okay, so where we left it is that the trolley problem's about to get real. Here's how Josh Green put it.
- [26:44]
C
You know, now, as we're entering the age of self driving cars, this is like the trolley problem now finally come to life.
- [26:51]
A
Oh, there's cars coming.
- [26:53]
D
Oh, the future of the automobile is here.
- [26:56]
A
Oh, there's cars.
- [26:57]
D
Autonomous vehicles.
- [26:59]
A
It's here.
- [27:01]
B
First self driving Volvo will be offered.
- [27:03]
C
To customers in 2020.
- [27:08]
B
Oh, where's it going?
- [27:09]
A
This legislation is the first of its kind focused on the car of the future that is more of a supercomputer on wheels. Oh, is the car coming?
- [27:17]
B
Okay, so self driving cars, unless you've been living under a muffler, they are coming. It's going to be a little bit of an adjustment for some of us. Hit the brakes, hit the brakes. But what Josh meant when he said it's the trolley problem come to life is basically this.
- [27:33]
C
Imagine this scenario, the self driving car now is headed towards a bunch of pedestrians. In the road, the only way to save them is to swerve out of the way. But that will run the car into a concrete wall and it will kill the passenger in the car. What should the car do? Should the car go straight and run over, say, those five people, or should it swerve and kill the one person?
- [27:54]
B
That suddenly is a real world question.
- [28:03]
C
If you ask people in the abstract.
- [28:04]
B
Like what theoretically should a car in this situation do?
- [28:08]
C
They're much more likely to say, I think you should sacrifice one for the good of the many. They should just try to do the most good or avoid the most harm.
- [28:15]
B
So if it's between one driver and five pedestrians, logically it would be the.
- [28:18]
A
Driver, kill the arm, driver, be selfless.
- [28:21]
B
I think it should kill the driver. But when you ask people, forget the theory.
- [28:27]
C
Would you want to drive in a car that would potentially sacrifice you to save the lives of more people in order to minimize the total amount of harm? They say, no. I mean, I wouldn't buy it.
- [28:36]
A
No, absolutely not.
- [28:38]
B
That would kill me in it. No.
- [28:41]
A
So I'm not going to, I'm not going to buy a car that's going to purposely kill me. Hell no, I wouldn't buy it for sure.
- [28:48]
B
No.
- [28:50]
A
I'll sell it, but I won't buy it.
- [28:54]
B
So there's your problem. People would sell a car and an idea of moral reasoning that they themselves wouldn't buy. And last fall, an exec at Mercedes Benz, face planted right into the middle of this contradiction. Welcome to Paris, one of the most.
- [29:15]
A
Beautiful cities in the world.
- [29:16]
B
Hello world and welcome to the 2016 Paris Motor Show. Home to some of the most beautiful.
- [29:20]
A
Cars in the world.
- [29:20]
B
Okay, October 2016, the Paris Motor Show. You had something like a million people coming in over the course of a few days. All the major carmakers were there.
- [29:29]
C
Here is Ferrari.
- [29:30]
A
You can see the Laferrari Alberta and.
- [29:33]
C
Of course the new GT4C Lusso T.
- [29:36]
B
Everybody was debuting their new cars and one of the big presenters in this whole affair was this guy.
- [29:42]
A
In the future you'll have cars where you don't even have to have your hands on the steering wheel anymore.
- [29:46]
C
But maybe you watch a movie on.
- [29:48]
A
The head up display or maybe you.
- [29:49]
C
Want to do your emails.
- [29:51]
A
That's really what we are striving for.
- [29:52]
B
This is Christoph von Hugo, a senior safety manager at Mercedes Benz. He was at the show sort of demonstrating a prototype of a car that could sort of self drive its way through traffic.
- [30:02]
A
In this E class today, for example, you've got a Maximum of comfort and support systems. You'll actually look forward to being stuck in traffic jams, won't you? Of course. Of course.
- [30:11]
B
He was doing dozens and dozens of interviews through the show. And in one of those interviews, unfortunately this one we don't have on tape, he was asked, what would your driverless car do in a trolley? Problem type dilemma where maybe you have to choose between one or many. And he answered, quote, if you know.
- [30:29]
A
You can save one person, at least save that one.
- [30:34]
B
If you know you can save one person, save that one person, save the.
- [30:37]
A
One in the car.
- [30:38]
B
This is Michael Taylor, correspondent for Car and Driver magazine. He was the one that Christoph von Hugo said that to.
- [30:45]
D
If you know for sure that one.
- [30:47]
A
Thing, one death can be prevented, then that's your first priority.
- [30:52]
B
Now when he said this to you, this is producer Amanda Ronczyk. Did it seem controversial at all in the moment?
- [30:57]
A
In the moment it seemed incredibly logical.
- [31:00]
B
I mean, all he's really doing is saying what's on people's minds, which is that.
- [31:05]
C
No, I mean, I wouldn't buy it.
- [31:07]
B
Who's going to buy a car that chooses somebody else over them anyhow? He makes that comment, Michael prints it and a kerfuffle ensues.
- [31:19]
A
Save the one in the car. That's Christoph von Hugo from Mercedes. But then when you lay out the questions, you sound like a bit of a heel because you want to save yourself as opposed to the pedestrians.
- [31:28]
B
Doesn't it ring though of like just privilege? It does, yeah, it does.
- [31:33]
A
Wait a second, what would you do? It's you or a pedestrian. And it's just, you know, I don't know anything about this pedestrian. It's just you are a pedestrian, just a regular guy walking down the street.
- [31:42]
C
Screw everyone who's not in a Mercedes. And there was this kind of uproar about that. How dare you drive these selfish, you know, make these selfish cars. And then he walked it back and he said, no, no, what I mean is that just that we have a better chance of protecting the people in the car. So we're going to protect them because they're easier to protect. But of course, you know, there's always going to be trade offs.
- [32:04]
A
Yeah.
- [32:04]
B
And those trade offs could get really, really tricky and subtle because obviously these cars have sensors.
- [32:15]
A
Sensors like cameras, radars, lasers and ultrasound sensors.
- [32:21]
B
This is Raj Rajkumar. He's a professor at Carnegie Mellon.
- [32:24]
A
I'm the co director of the GM CMU Connected and Autonomous Driving Collaborative Research Lab.
- [32:29]
B
He is one of the guys that is writing the code that will go inside GM's driverless car. He says, yeah, the sensors at the.
- [32:36]
A
Moment on these cars still evolving.
- [32:39]
B
Pretty basic.
- [32:39]
A
We are very happy today. We can actually detect a pedestrian, can detect a bicyclist, a motorcyclist, different vehicles of different shapes, sizes and colors.
- [32:48]
B
But he says it won't be long.
- [32:50]
A
Before you can actually know a lot more about who these people are.
- [32:53]
B
Eventually they will be able to detect people of different sizes, shapes and colors. Like, oh, that's a skinny person, that's a small person, tall person, black person, white person, that's a little boy, that's a little girl. So forget the basic moral math, like what does a car do if it has to decide, oh, do I save this boy or this girl? What about two girls versus one boy and an adult? How about a cat versus a dog? A 75 year old guy in a suit versus that person over there who might be homeless. You can see where this is going and it's conceivable the cars will know our medical records. And back at the car show, we've.
- [33:25]
A
Also heard that term car to car communication. Well that's also one of the enabling technologies in highly automated driving.
- [33:32]
B
Mercedes guy basically said in a couple years the cars will be networked, they'll be talking to each other. So just imagine a scenario where like cars are about to get into accidents and right at the decision point they're like conferring, well, who do you have in your car? Me, I got a 70 year old Wall street guy, makes eight figures. How about you? Well, I'm a bus full of kids. Kids have more years left. You need to move. Well, hold up. I see that your kids come from a poor neighborhood and have asthma. So I don't know.
- [33:55]
A
So you can basically tie yourself up in knots, wrap yourself around an axle. We do not think that any programmer should be given this major burden of deciding who survives and who gets killed. I think these are very fundamental, deep issues that society has to decide at large. I don't think programmer eating pizza and sipping coke should be making that call.
- [34:22]
B
How does society decide? I mean, help me imagine that.
- [34:25]
A
I think it really has to be an evolutionary process.
- [34:27]
B
I believe Raj told us that two things basically need to happen. First, we need to get these robocars on the road, get more experience with how they interact with us human drivers and how we interact with them. And two, there need to be like industry wide summits.
- [34:41]
A
No one company is going to solve that.
- [34:43]
B
This is Bill Ford Jr. Of the Ford Company giving a speech in October of 2016 at the Economic Club of.
- [34:50]
A
D.C. and we have to have. Because could you imagine if we had one algorithm and Toyota had another and General Motors had another? I mean, it would be, I mean.
- [34:58]
B
Obviously couldn't do that because, like, what if the Tibetan cars make one decision and the American cars make another?
- [35:04]
A
So we need to have a national discussion on ethics, I think, because we've never had to think of these things before, but the cars will have the time and the ability to do that.
- [35:18]
B
So far, Germany is the only country that we know of that has tackled this head on. One of the most significant points the ethics commission made is that autonomous and connected driving is an ethical imperative. The government has released a code of ethics, says, among other things, self driving cars are forbidden to discriminate between humans in almost any way. Not on race, not on gender, not on age, nothing. These shouldn't be programmed into the cars.
- [35:47]
A
One can imagine a few classes being added in the Geneva Convention, if you will, of what these automotive vehicles should do. A globally accepted standard, if you will.
- [35:58]
B
How we get there to that globally accepted standard is anyone's guess. And what it will look like, whether it'll be like a coherent set of rules or like rife with the kind of contradictions we see in our own brain, that also remains to be seen. But one thing is clear.
- [36:15]
A
Oh, there's cars coming. Oh, oh, there's cars.
- [36:21]
B
Oh, there are cars coming. With their questions.
- [36:25]
A
Put me back for me. Control it.
- [36:28]
B
Oh, dear Jesus.
- [36:29]
A
I could never. Ah. Ah.
- [36:32]
B
Oh, where's it going? God damn, Bill.
- [36:36]
A
Oh my God.
- [36:38]
B
Okay, we do need to caveat all this by saying that the moral dilemma we're talking about in the case of these driverless cars is going to be super rare. Mostly what will probably happen is that like the plane loads full of people that die every day from car accidents, well, that's just gonna hit the floor. And so you have to balance the few cases where a car might make a decision you don't like against the massive number of lives saved.
- [37:05]
A
I was thinking actually of a different thing. I was thinking even though you dramatically bring down the number of bad things that happen on road, you dramatically bring down the collisions, you dramatically bring down the mortality. You dramatically lower the number of people who are drunk coming home from a party and just ram someone sideways and killing three of them and injuring two of them for the rest of their lives. Those kinds of things go way down. But the ones that remain are engineered like they are calculated almost with foresight. So here's the difference and this is just an interesting difference. Oh, damn, that's so sad that happened that that guy got drunk and da da, da. Maybe he should go to jail. But you mean that the Society engineered this in. That is a big difference. One is operatic and seems like the forces of destiny, and the other seems mechanical and pre thought through.
- [38:06]
B
Premeditated.
- [38:06]
C
Yeah.
- [38:07]
A
And there's something dark about a premeditated expected death. And I don't know what you do about that, but everybody's on the horizon.
- [38:14]
B
In the particulars. In the particulars, it feels d. It's a little bit like when you know, should you kill your own baby to save the villains? Like in the particular instance of that one child, it's dark. But against the backdrop of the lives saved, it's just a tiny pinprick of darkness. That's all it is.
- [38:28]
A
Yeah, but you know how humans are. If you argue back that, yes, a bunch of smarty pantses concocted a mathematical formula which meant that some people had to die. And here they are. There are many fewer than before. A human being, just like Josh would tell you, would have a roar of feeling and of anger in saying, how dare you engineer this in. No, no, no, no, no.
- [38:51]
B
And that human being needs to meditate like the monks to silence that feeling, because the feeling in that case is just getting in the way.
- [38:58]
A
Yes and no. And that may be impossible unless you're a monk, for God's sake.
- [39:05]
B
See, we're right back where we started now. All right, we should go.
- [39:10]
A
Chad. You have to thank some people. No.
- [39:11]
B
Yes. This piece was produced by Amanda Aranchak with help from Bethel Hobte. Special thanks to Iyad Rawan, Edmond Awad and Sydney Levine from the Moral Machine group at mit. Also thanks to Sirtac Karaman, Xin Chang and Roborace for all their help. And I guess we should go now.
- [39:28]
A
Yeah, I'll.
- [39:31]
B
I'm Jad Abumrad.
- [39:32]
A
I'm not getting into your car, if you don't mind. Just take my own.
- [39:37]
B
I'm gonna rig up an autonomous vehicle to the bottom of your bed. So you're gonna go to bed and suddenly find yourself on the highway driving you wherever I want.
- [39:48]
A
No, you won't.
- [39:50]
B
Anyhow. Okay, we should go. I'm Jad Abumra.
- [39:52]
A
I'm Robert Kulwich. Thanks for listening. Received today at 2:41pm all right, this.
- [40:19]
C
Is Josh Green giving you your credit.
- [40:21]
D
Hi, this is Michael Taylor for Amanda.
- [40:23]
A
Here we go.
- [40:24]
C
Radiolab was created by Jada Bumrad and is produced by Soren Wheeler.
- [40:28]
A
Dylan Keith is our director of sound design.
- [40:31]
C
Our staff includes Simon Adler, Rachel Rachel.
- [40:33]
A
Cusick, David Gabble, Tracy Hunt, Matt Kielke, Robert Crowdy. Sorry, I'm in an airport and we've.
- [40:41]
C
Got the the overhead announcement. All right, I'll keep going.
- [40:48]
A
Matisse Master, Melissa o', Donnell, Harry Ann Wack and Molly Webster.
- [40:53]
C
With help from Amanda Aranchic, Shima Oli, David Fox, Nigel Fatali and Nigar Fatali, Phoebe Wang and Katie Ferguson.
- [41:03]
A
Our fact checker is Michelle Harris. I hope that does it for you.
- [41:06]
C
Let me know if you want me to do it again. Thanks, guys.
- [41:08]
A
I'm looking forward to hearing the show. Bye. End of message. With Venmo Stash, a taco in one hand and ordering a ride in the other means you're stacking cash back. Nice. Get up to 5% cash back with Venmo Stash on your favorite brands when you pay with your Venmo debit card. From takeout to ride shares, entertainment and more, pick a bundle with your go tos and start earning cash back at those brands. Earn more cash when you do more with Stash. Venmo Stash terms and Exclusions apply. Max $100 cash back per month. See terms at Venmo Me StashTerms for delicious meals, you could go out to.
- [41:48]
B
Eat, or you could just make a Marie Callender's meal.
- [41:50]
A
Marie Callender's classic Chicken Parmigiana bowl is so good.
- [41:54]
B
It has marinara sauce that's made from scratch and creamy mozzarella cheese over pasta.
- [41:59]
A
It's delicious with no artificial flavors, colors.
- [42:02]
B
Or preservatives and 30 grams of protein. You can find it in the frozen aisle. Marie Calendars what having it all tastes like.