AI Avatar of Killed Man Testifies in Court - The 404 Media Podcast

Loading summary

Transcript65 lines

[00:01]
Kate
The other words of the year, they were just as online, you know, Oxford's was brain rot, if you can believe it. Its first use was in 1854 by Henry David Thoreau.
[00:12]
Joseph
RIP Henry David Thoreau.
[00:14]
Sam
You would have hated TikTok.
[00:17]
Joseph
Wait, Candace, we should probably introduce ourselves.
[00:20]
Kate
You're so right, Kate. Hello, listeners. We are the hosts of icymi, or rather, in case you missed it, Slate's podcast about Internet culture where we're extremely.
[00:28]
Joseph
Online so you don't have to be follow ICYMI now, wherever you listen. Hello and welcome to the 404 Media podcast where we bring you unparalleled access to hidden worlds, both online and IRL. 404Media is a journalist, founder, company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content@404me co. I'm your host, Joseph, and with me are 404 Media co founders Sam Cole.
[01:09]
Sam
Hey.
[01:10]
Joseph
Emmanuel Mayberg.
[01:11]
Emmanuel
Hello.
[01:12]
Joseph
And Jason Kebler.
[01:14]
Kate
What's up?
[01:16]
Joseph
Let's go straight into one that you wrote, Jason, along with Matthew Golt, our regular contributor. The headline is, I loved that AI judge moved by AI generated avatar of man killed in road rage incident. Let me just straight away play some of the audio and video, if you're watching on YouTube, of this avatar. It's just a few seconds, I think.
[01:45]
Jason
Hello. Just to be clear, for everyone seeing this, I am a version of Chris Pelke recreated through AI that uses my picture and my voice profile. I was able to be digitally regenerated to share with you today. But here is insight into who I actually was in real life. Take a look.
[02:05]
Joseph
Then the video carries on and it shows actual footage of Chris with family members and friends. If I recall correctly, Jason, what did we just see in here?
[02:18]
Kate
Yeah, yeah. And then at the end, let's play one more little bit where he basically forgives the man who killed him, which is very odd or at least was very weird to watch. But yeah, this is Christopher Pelke. He's a man in Arizona who was killed in a road rage incident by a man named Gabriel Horquesitas. And what you're hearing is the end, the very, very, very, very end of the trial where Horquisitas is being sentenced. And so basically he had already been found guilty. And this is a sentencing hearing where as part of it, everyone is allowed to submit a statement more or less like a victim testimony, more or less. And so what this means is people who know Gabriel Horquacitas can you know, submit like testimony saying this was a one time incident, like he's a good family man, things like that, that like that. Those are sort of the other types of statements that were submitted. And then Christopher Pelke's family was like, you know, he was such an amazing person. We're very sad that he's dead. Of course, like this is just a normal part of sentencing. And then the judge considers all of that and comes up with how long the person should be in prison.
[03:48]
Joseph
So it happens every single day in cases all across the states. Very, very standard. And usually doesn't involve AI Usually doesn't.
[03:57]
Kate
Involve AI And I guess what I'm saying is like this AI was not cross examined or anything like that. Like this is a pre recorded statement and we'll get into how it was made. But the, this was submitted to the court as essentially Chris Pelke's quote unquote own statement. So in this case you had a man who was killed who from beyond the grave submitted a statement about who he was as a person and at the end like directly addresses the killer and says, you know, like I, I forgive you. Which is, it's just very wild to think about. I, I think that this is very much like, I think this trope is over way overused. But this is very Black mirror to me. Like this is the type of things that Black Mirror examined, especially early on in its run where it's like you'll be able to digitally recreate people after they die and they'll, you can like continue to have a relationship with them. In it was a man who was killed who is testifying at the trial for, I mean it ended up being a manslaughter, but testifying at the trial for the man who killed him.
[05:20]
Joseph
Yeah. And maybe here this is where we can just drop some more audio and video of that latter part of the video where as you say, he sort of directly addresses his killer to Gabriel.
[05:34]
Jason
Horquesitas, the man who shot me. It is a shame we encountered each other that day in those circumstances, in another life we probably could have been friends. I believe in forgiveness and in God who forgives. I always have and I still do.
[05:52]
Joseph
So how do we get. Well actually no, I was going to save this for later, but I almost just want to immediately address it. The family says this is almost like the victim impact statement from Christopher. That is not true. Right. It is a creation that's been made and we'll get into that. In a second by the family. And I'm of course not even talking about this family's grief or what they must be feeling. I just mean on almost a factual, cold, legal basis, it's not him talking. Right? I mean, what do you make of that?
[06:32]
Kate
Yeah, I mean, I think that this case is a little bit, you know, it's really weird because you have a grieving family and I don't want to downplay that at all, but you have a grieving family that's trying to figure out how to cope with this situation. And basically his sister, who we spoke to a woman named Stacey Wales, she wrote this. Like this was actually not. The actual words were not generated by an LLM that was trained on Christopher Pelkey or anything. Like she wrote his words or quote unquote his words and then her husband. So Christopher Pelke's brother in law is the one who recreated his voice. So they use an LLM to like train. Train on his voice and recreate his voice. And then they also used Elora, which Emmanuel has talked about a lot, is basically a mechanism for like recreating a, a person from an image. And so that is. That is what, like how this was made?
[07:42]
Joseph
Yeah, I mean, Emmanuel, could you maybe talk a little bit more about that tool just for a second? Like, is it an app? Is it a tool? Is it just free to use? Does it just require one image? What's the deal with it?
[07:54]
Emmanuel
I would really define it as a method most commonly used with stable diffusion, which is an open weights AI model that can do all sorts of things. People know it mostly from its ability to generate images. And essentially it's just a way of very quickly training stable diffusion to recreate the likeness of a person or an object or a scene or a style. It's a way to like training Stable diffusion took a lot of time and a lot of training. Data and allure is a way to use just a few images or inputs to make it do something very specific that it wouldn't be able to do otherwise.
[08:37]
Joseph
Gotcha. That makes sense. So very accessible. It sounds like, you know, basically anybody could use it. And I presume that's why, you know, somebody like a member of the public or whoever may turn to it. Jason, what did. Well, two things. Why did Christopher's sister want to make this? And then second of all, sort of what did they make of the result? Were they happy with it? Or so the why and how good was it?
[09:06]
Kate
Yeah, I mean, it sounds like she did this because she was having trouble writing her own victim impact statement for the trial. I mean, I sort of can't imagine being in this situation and knowing the words to say. And also, I mean, thankfully, I've never been in that situation, but sort of knowing like, what to even ask for or how to feel about it. It should be noted that the family actually asked for the maximum sentence time because the family was able to submit, like, what they wanted. But interestingly, like, Christopher Pelke's AI statement said that he forgives the person who killed him. And he didn't specifically. And I say he. But obviously this video doesn't specifically ask for anything. But the tone of it is one that's sort of like, it's very unfortunate that this happened. He says at one point, you know, in a different life, we could have been friends. And it's kind of hard to know, like, what to make of the whole thing, I guess.
[10:11]
Joseph
And like, truth wise, like, is that really what he would have thought? Like, what do you mean?
[10:16]
Kate
Well, I can talk more about that. I mean, I think that there's no way of knowing what he would have thought because it's just regardless of how well, you know someone, I think that, you know, I have lost someone who was very, very close to me. And the first thing that happened after she died was all of these people who barely knew her started saying, oh, she would have loved this. She would have said that she would have done this. And first of all, like, she wouldn't have done most of those things. Like, I know for a fact. But also you can never know. You can never know what someone. And that, that's one of the, like, really awful things about death. And I think that's actually, that's actually like really important to this case because this is fiction. It is fiction masquerading as a legal document. Because you have no way of knowing what Chris Pelke would have said or would have done because he is dead. And you just simply can't know that no matter how close you were to him. And so you essentially have like his sister who has written something for him and really, like, it's masquerading. I mean, people have called it a puppet. Like, this story went pretty viral. Not just our version, but it was widely covered in the press. And you just can't know.
[11:40]
Joseph
Yeah, because some other people covered it because the videos came out. And I'm gonna play another one in a second. And then we spoke to the sister and I got more detail. But you did mention, you know, kind of framing this as a legal document and it is literally like it's been entered into court. And this AI avatar has, in a legal sense, testified in this hearing, someone who should know about legal documents and the sorts that should be allowed and potentially shouldn't be allowed, you know, are judges. And here is the judge's reaction to seeing this AI played in their courtroom. I love that AI.
[12:27]
Kate
Thank you for that. And as angry as you are and justifiably angry as the family is, I heard the forgiveness, And I know Mr. Horticides appreciate it, but so did I.
[12:42]
Joseph
So the audio was not the best quality there, but basically the judge says that I loved the AI and the family could have appreciated it, but so did the judge. Some of the coverage didn't really focus on the judge's reaction. Of course, it's in the headline of ours because it's pretty important. What did you make of the judge's reaction when you saw it, Jason?
[13:08]
Kate
Yeah, I mean, I was really surprised that it was accepted into the court. And, you know, I'm not a lawyer, especially not a. Well, I'm not any kind of lawyer, but I have less familiarity with the criminal legal system than I do some of some of the, like, other types of lawsuits that we cover all the time regarding, like, lawsuits and privacy. Yeah, copyright. But I. I know very little about the proceedings of, like, a criminal court, but I have to imagine that judges have pretty wide leeway to determine what can be accepted and what cannot. So I was shocked personally that he accepted this statement and said that he liked it. And I think it really does raise questions about how well informed judges, lawyers, victims, suspects, et cetera, are about how AI tools work. And we know that, like, law firms are sort of foaming at the mouth to use AI tools because it can really cut down on the amount of time that it takes to do things, to go through, like, really big legal documents. Like, you know, Sam has written several articles about lawyers who have, like, written documents for the court that hallucinated, you know, other court cases because they used an LLM. And in some cases, like, the judges chastised the lawyers for doing so. And, you know, in this case, the judge didn't chastise the family for doing this. He said, thank you for doing this, and. And accepted it. I mean, I think Sam can talk a little bit more about how the legal profession is using AI But I don't know, I think it opens up a lot of questions, like, can you. Can. Can a victim's family call an AI chatbot trained on someone's text messages to you know, testify in a court, like against. In a murder case. Like, I have no idea. Like, it raises up. It raises all sorts of questions to me.
[15:23]
Joseph
Yeah. Sam, what do you think? On one side, we have the judge here embracing it, and then you've covered all of these cases where lawyers have basically effed up because they've been using AI in their own sort of writings. What do you make of it?
[15:38]
Sam
I mean, I thought that was a big part of why this was such a strange and kind of shocking story. A lot of judges are threatening lawyers with fines and fees and severe punishment for using LLMs in compiling documents for courts. Like, we see a couple times. Most recently, I think the MyPillow attorneys did this, but we've covered it a couple times where lawyers are compiling cases and they're citing other cases that are comparable. Like, you know, see this case, that case, to kind of hold up their own argument. And either the opposing side or the judge catches that the cases are just not real. They don't exist. That's because they're using AI. Who knows which one? There's many. But, for example, they're using ChatGPT. You can imagine them kind of typing in, make some cases, cite some cases for me as part of this case that I'm working on, and then they plug that straight into a court document and submit it. And that's really sloppy, and it would piss me off as someone who hired that lawyer. And it pisses off the judge because the judge is like, this is a waste of time. This is a mockery of the profession that you're in. So to see this judge in particular say, be so moved by it is just. I don't know. It's like, you went to law school, bro. Like, what are you talking about? This is purely the conjecture of this guy's sister. It's not anyone's words but hers and her husband's and the family's. So it's just. It was amazing to me that the judge was so. I don't know, I don't want to say gullible, but, like, that's kind of what it is. And then they were asking for the full sentence. So, like, really, was he so magnanimous as to forgive his killer?
[17:41]
Kate
It should be noted that the judge gave the killer the maximum sentence as the family asked for. So it's like, it's pretty. The. The forgiveness from the AI avatar did not sway the judge to go more lenient.
[17:57]
Sam
And it wasn't meant to.
[17:58]
Kate
It wasn't meant to because it came from the family. And the. The sister told us. Here's what she told us. She said, quote, our goal was to make the judge cry. Our goal was to bring Chris to life and to humanize him. And it's like, I think you can feel a lot of potentially different ways about it, because I'm sure that any family that's been through this will say that they feel like they've like, lost their voice. Like, Chris cannot speak for himself. He cannot speak to what happened that day. But that is just like how life is. It's like it's the definition of. Of a murder or, or any, like, any case where someone dies. It's like they cannot describe what happened to them.
[18:44]
Emmanuel
I just. Yeah, I want to note that there's a long history of people more likely to put their critical thinking aside where new technology meets death. And I think there. That's because there's such a. A longing and like the unknown about death that people accept really wild ideas. That's something that I covered a few times when we were at Motherboard. So, for example, in 2015, when VR was all the rage, there were quite a few projects that claimed to let you connect with your dead loved ones by creating some sort of VR experience that recreate the likeness of a lost parent or something and having some voice acting in there and that letting you communicate with the dead, which, again, obviously is not true. There have been a lot of kind of Holocaust memorial project that use holograms and other new technologies, which, again, are like, well intentioned, as I think the family is in this case. But as Jason said, it's off. You're not communicating with someone who has died. It is more like talking to a psychic, which I think as a culture we're more likely to acknowledge is a scam. But for some reason, when you put the label of technology on it, it is something that you can present seriously in a court. And with AI being the hot new technology, we've seen a ton of this stuff around death. I will remind people that Sam covered this company called Replica many times and that it is essentially an AI girlfriend app. It is a companionship AI model. But it grew out of this project where someone created an AI model of their dead partner and that kind of had this huge glowing feature in the Verge. And again, it was seriously engaging with this idea that this allowed someone to talk to the dead, and then five years down the line is just a sex robot of some kind of. So, yeah, just. Just like this is a thing that continues to happen over and over. Again, but it is definitely not something you'd see admissible in court, usually.
[21:27]
Kate
Yeah, I mean, you're absolutely right. And I would just add that one of the core projects of AI in general is to recreate consciousness. Like that is the, the goal of it. When we talk about, you know, super intelligence and the singularity and things like that, it's like to create a conscious being and we don't need to dive into it now. It's, there's many reasons to be very skeptical of this, but that's the entire selling point. That's like behind all the AI safety. And then there's a lot of, you know, transhumanists and like AI developers who specifically want themselves to be, to live on as an AI. They want their own consciousness to be like, uploaded into the everlasting cloud. And so I think that that is very often. There's so many AI startups that have this as their goal and just to sort of transcend death in some way. And there's been many, many projects like this. And so, yeah, to see it in a court, I mean, when I saw it, I was shocked. And it's like, I'm very rarely shocked by anything with AI anymore, but I'm like, oh, you've gotta be kidding me. Like, I cannot believe that this was accepted.
[22:46]
Joseph
Yeah, all right, maybe we'll leave that there. When we come back after the break, we're going to talk about our interactions with Meta and I suppose Silicon Valley PR reps more broadly and specifically, it's about Meta's plans to put AI, sorry, facial recognition, same sort of thing, into its smart Ray Bans. We'll be right back after this.
[23:26]
Kate
For years I've just been trimming my beard because I found shaving to be uncomfortable and razors to be expensive. Then I realized that I just didn't have the right tech. Staying well groomed requires the finest and most advanced tools. Upgrade with Harry's. Get the best razors money can buy for a fraction of the price of bigger brands. Right now you can get a $10 trial set for just $5 at Harry's.com 404, which is exclusive to our listeners. Like I said, I haven't regularly shaved my face for years because I have sensitive skin and didn't like dealing with the fuss. Then I got Harry's trial set, which comes with a premium five blade razor, foaming shave gel with aloe, which is great for sensitive skin, and a travel cover. I had the best shave I've had in years. Totally smooth and no razor bumps, which is huge for me. Harry's has the highest customer satisfaction in the shaving industry, a no risk trial and a convenient subscription model that you can cancel at any time. Don't like your shave? No worries, it's on them. Take your grooming out of the stone Age. Get Harry's Normally their trial set is ten dollars, but right now you can get it for just five dollars at Harry's.com404 that's our exclusive link Harry's.com404 for a five dollar trial set starting your own business is hot this summer, but lots of us don't vacation alone. And you shouldn't dive into starting a new business alone either. It goes a long way to have proven experts to help you navigate that great unknown. And sometimes it makes sense to have a tour guide. For millions of businesses, Shopify is their guide. A tool that simplifies and supercharges your business. Shopify is the commerce platform behind millions of businesses around the world and 10% of all E commerce in the US. It's also the tool we use for our own merch store. Setting up and managing our store couldn't have been easier thanks to Shopify's powerful, user friendly interface. Get the word out like you have a marketing team behind you. Easily create email and social media campaigns wherever your customers are scrolling or strolling and get started with your own design studio. With hundreds of ready to use templates, Shopify helps you build a beautiful online store to match your brand style. And best yet, Shopify is your commerce expert with world class expertise in everything from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into With Shopify on your side, sign up for your $1 per month trial and start selling today at shopify.com media. Go to shopify.com media shopify.com Summer is just around the corner and the folks at Mint Mobile have a hot take. Getting a summer bod is out and getting your savings BOD is in. This spring and summer we want skimpy wireless bills and fat wallets. And with premium wireless plans for just 15 bucks a month you can have both without breaking a sweat or the bang. Switching to Mint Mobile is a breeze. Soon you'll be getting great wireless coverage on vacation, at the beach or at the lake. Say bye bye to your overpriced wireless plan's jaw dropping monthly bills and unexpected overages. Mint Mobile is here to rescue you. All plans come with High speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of premium wireless service for Mint Mobile for 15 bucks a month. This year. Skip breaking a sweat and breaking the bank. Get your summer savings and shop premium Wireless plans@mintmobile.com 404 Media that's mintmobile.com 404 Media Upfront payment of $45 for 3 month 5 gigabyte plan required equivalent to $15 per month. New customer offer for first 3 months only. Then full price plan options available, taxes and fees extra. See Mint Mobile for details.
[27:51]
Sam
Well, well, well, we're back. Well, well, well, Joseph. What is going on here? So, Joe Rosery. The headline is well, well, well, Meta to add facial recognition to glasses after all. So if you're a longtime reader of 404, you know what we're alluding to already. But a while back, how long ago was it, Joe? Was it like maybe earlier this year or last year?
[28:25]
Joseph
I believe it was last October. October 2024.
[28:28]
Sam
Okay. So yeah, in October, these two Harvard students did something pretty wacky with Meta's. They have these Ray Ban glasses that you can wear and they have cameras in them. So these two Harvard students got a hold of these glasses and did something with them. So do you want to walk us through that original story first before we get into the smugness of it all?
[28:50]
Joseph
Yeah. So they take these Meta Ray Ban smart glasses. I always change the order of those words every time I say them. But they took those and they have the camera on board and then they basically strung together all of these different technologies which would include PIM Eyes, which is the off the shelf facial recognition tool basically anybody can use. People have used it to identify January 6th rioters. People have used it to dox sex workers. People have used it in all sorts of abusive and then somewhat interesting ways as well. But they paired basically PIM Eyes with these Ray Ban glasses and they called it Eye X Ray, if I'm remembering correctly. But it would also go a step further in that PIM Eyes gives you the webpage where that matching face was found online. So let's say you're looking at somebody with the glasses. It captures their face, it then puts their face through PIM Eyes and it brings up, let's say, the website of the law firm that this person works at. Then we could take the name attached to that, put that through an LLM and look for more information. And the idea was that it could basically dox somebody just by looking at them, finding where they live, finding where they work, all of this other stuff, and puts that all together into one package. And they made this video, which went viral on Twitter. X of the two students using it, they said, on dozens of people without their knowledge, like unsuspecting, and I would say victims, maybe some people were okay with it, but just going around and revealing the identities of people who were anonymous members of the public. And this is the big, big fear when it comes to putting not just facial recognition tech in the hands of anybody, but facial recognition tech in the hands of anybody in a surreptitious manner, in that I can just look at somebody and my Google Glass from a million years ago or now my meta ray bans are going to reveal to me who that person is. Where ordinarily they would be anonymous and they just walk down the street. I'm never going to see them again. Now I know who they are and I know everything about them. And I mean, it's a terrifying proof of concept. So we covered that at the time and we'll get into more detail in a second. But Meta pushed back a lot against that article and they didn't really like that we were reporting on it.
[31:34]
Sam
The wording that you used in the story that we're talking about today is they let me get it right. They chewed my ass off.
[31:43]
Joseph
Yes. That's a technical term.
[31:46]
Sam
Yeah. So with that context in mind, what is meta announcing now as of last week?
[31:55]
Joseph
Yeah. So there's a report in the Information that meta is actually working on facial recognition tech for its meta ray bans. And like, we don't know all that much, but basically they're working on it actively to put that capability into the glasses. And they're also doing, or they're also exploring this other thing, which is not having or potentially not having the light turn on on the glasses. So usually when you put the metal ray bans into recording mode, or whatever you want to call it, a little light will appear, which will indicate to other people, oh, hey, that person's recording. I don't think enough people wear these glasses for that to be common knowledge, to be perfectly honest. Like, you know, if someone's shoving an iPhone in your face or a GoPro in your face, you don't know the little red light on the pair of ray bans means you're being filmed. Regardless, Meta is reportedly looking at getting rid of that as well, so after all that coverage we did as students and have Meta's little meltdown about it. Well, well, well, they're putting facial recognition into the glasses themselves anyway.
[33:04]
Sam
Yeah. And they were super pissed when the first article with the students came out because obviously these are really, this is like a really privacy invasive thing to have people walking around doing facial recognition in real time to everybody around them without their consent. I don't know, I find that to be a pretty dark world to imagine living in. But Meta was like, you know, they were, I mean, walk us through what they said in response to your first story because they were pretty upset with like every element of the story. Yeah, like we said, they really did chew your ass off. I mean, they basically back and forth in emails.
[33:50]
Joseph
Yeah, they basically disagree with the premise. And I won't go into every single sentence they wrote. You can go and read the article for that, but I'll give like the summary of it when I reached out for comments saying, hey, look, these students have taken your technology made by Meta and Ray Bans, I suppose as well, and they've paired it with this facial recognition technology and these people search databases. Can I have your comment? Blah, blah, blah. Because we're writing this up now. And they initially asked, hey, just to clarify, this could be done with any sort of camera, Right. It's not something that's intrinsic or inherent to the Meta Ray bans. You could do this with a stationary camera or whatever, and then you've strung those different technologies together as well. I mean, yes, that is true, but as I explained to the Meta spokesperson, the students didn't strap a GoPro to their helmet or their face and then walk around trying to dox people, because that would be really, really obvious what they were doing. And again, by the student's own omission, the entire point was to do this in a stealthy manner and to do it on unsuspecting people. If you're shoving an iPhone in somebody's face, that's not going to be unsuspecting. We're all used to that by now. So they take issue with that. And I explained it. Whatever. They then take an issue with the headline, which we did, which was, someone put facial recognition tech onto Meta's smart glasses to instantly dox strangers. They had some sort of issue with the word onto Meta's smart glasses, where they're like, oh, you're suggesting that the glasses were modified in some way. And I'm like, I don't know, man. Onto into, slapped on, shoved together. It doesn't matter, man. It's completely meaningless distinction. And the impact is that this technology is being used in this way. We then record a podcast about it that people can go back and listen to if they want to get much more detail there. Headline of that was the smart glasses that docs strangers. And then the response was, and I think I'm just going to read this out. This is the only section I'm going to read out, but it kind of gives you insight into Silicon Valley press reps. And they said you say in the podcast that the glasses don't have facial recognition capabilities. And you've previously acknowledged that this could be done with any camera or recording device. But a headline saying smart glasses that dock strangers clearly makes it sound like this is the issue that is specific to the glasses, or that the facial recognition was executed on the glasses themselves versus the reality, which is that this was all run by a program on their laptop. This is despite the fact that students themselves have said publicly, we do not want this to be a criticism of their product at all. And we just had them on hand. This could have been done on a phone camera. And he finishes the quote and says, I realise we may not agree on everything here, but surely you can appreciate how headlines like this are misleading for readers, as I explained in this well, well, well piece. No, I don't appreciate that whatsoever. And I don't care if the Harvard students were saying, well, we didn't want this to be a criticism of Meta's glasses. It's like, dude, you took them and you built an instant doxing device. It doesn't matter what your intention was. And in fact, that's actually more worrying. You've clearly built it, not fully understanding what you've actually unleashed here. And this is just what it's like dealing with Silicon Valley reps. And I guess that's something I actually just wanted to ask you, Sam and Jason Emmanuel as well. Is this what it's like interacting with PR reps from tech companies? Like, what. What do you all think of that?
[37:41]
Kate
Is that a rhetorical question?
[37:42]
Joseph
Yes, yes, it's literal. Because I just. Because I want you to fill it in. As in, is this an extreme. This isn't like the most extreme thing. It's just a guy moaning at me, an email, so who really cares? It's because they now announced they are doing facial recognition. That's important. But, like, I mean, what did you make of it, Jason?
[38:03]
Kate
I mean, I thought it was absurd. I thought it was absurd then and now I think, well, well, Well, I don't know what else to say other than that. But yes, this is, it's interesting because I think we try to be as transparent as we can about how we do our journalism and what, what that means, like what, where our information comes from and that sort of thing. And I think part of us doing our articles is going to the people and companies that the article is about and saying, well, what do you think of this? Like, give us a comment. Like, here's a thing that's happening, like what's, what's going on? And we actually very often at this point have to go to companies and say, give us a comment. Like, answer these questions. Here's a statement, here's a comment. And that's actually not the ideal. The ideal is like make someone in your company available for an on the record interview to discuss this situation so we can have a back and forth dialogue. And that happens pretty much never anymore. I mean, I can't remember when was.
[39:14]
Emmanuel
The last time, when was the last time?
[39:15]
Kate
I mean, it happens never. And, but, and when it does happen, it's like Mark Zuckerberg is talking to a friendly podcaster for three hours. He's like talking to Lex Friedman for three hours. But like, when is the last time that anyone tech company did like an on the record, hard hitting interview with a real journalist? Like, I don't know when, I don't know when and when they do it. It's to promote like a specific thing. And so it would be interesting to have this debate with Mark Zuckerberg or you know, we understand he's a, he's a busy person. So how about someone who's working on the Smart Glasses project? How about an on the record conversation with the PR person? And it's like in this case none of this was on background, which means you can use the information or sometimes it means you can quote it but not say who, who it came from. Like the definition of what on background even means is different for every person. And if you're listening to this, you have no real reason for knowing what that is. But basically it's like when you deal with spokespeople at companies, they usually immediately demand to go off the record, which we can either refuse to do, which is tricky, tricky.
[40:34]
Joseph
Sometimes we do depends on the story, but sometimes we do it.
[40:36]
Kate
Yeah, sometimes we do it. I mean, I think we try not to because it's like not ideal, but sometimes we do just because we're trying to be fair to them. We're trying to explain, explain what the story is, we're trying to. Sometimes they want to go off the record because something is like a giant misunderstanding and it's like not actually a story and like, this is rare, but sometimes it's like we have something wrong and they want to tell us, like, hey, this is like wrong. And if they were to say that on the record, then maybe we would do a story saying like, I don't know, it's just like, it's a tough thing to even explain.
[41:13]
Joseph
Facebook denies XYZ when it's like, it's straight up wrong. But as you say, that's super rare anyway.
[41:19]
Kate
Yeah. And so it's like, it's like trying to get blood from a stone, I guess. It's like getting information from companies through official channels is like, they're doing pr. We are sort of doing like due diligence, just being like, yo, like, here's what we're writing. What do you have to say about it? And it's become like a really antagonistic type of back and forth or it's become like a box that you tick. Because company, like companies broadly, they're like, we don't give a shit. Like, here's, here's some like canned statement that doesn't mean anything. And then we put it in our article because we sort of have to, because that's just like how it works. And it's like half the time the companies are lying is the wrong word, but half the time they are like trying to like split hairs so finely that it doesn't mean anything to any reader. And as you said, Joseph, it's like onto into, taped to like, who gives a vibes? And then the other times it's like they are. They are doing some sort of like gymnastics with their boss where their boss is like very mad that a specific word or a specific company is in the headline or something and that. And they're like willing to go to the mat with you to like demand that you change a headline because someone at their company doesn't like it. And it's like, we don't change it unless it's wrong. And if it's wrong, then we'll fix it. But I don't know. That's a lot. But it's. Dealing with these people is a nightmare.
[43:02]
Emmanuel
It seems to me like the last time we had anything resembling a revealing conversation with a comms person at a tech company was back when Joe and Jason did this big feature about Facebook global moderation. And that was in the wake of Facebook being blamed for inciting violence in Myanmar and the genocide there. And back then, it seems like the comm strategy or their response to that was like, we're really sorry this happened. This was really bad. Here's what we're trying to do. Here's why this happened, despite our best efforts. And now we've revealed information that helps you better understand the company and we've learned lessons and yada, yada, yada, and that's the story. And I feel like since then, especially since Zuckerberg's transformation into like this right wing pro Trump figure is the PR people are just pushing back for the sake of pushing back on any story that they don't like because it's like a game and they want to score some points. So whatever it may be, they just want like you wrote a negative story. It seems to me my read on their strategy as a whole, and this is true about many tech companies, is that they want some sort of pushback in the article. And it almost doesn't matter what it is. It can be their comment, it can be a denial, it can be getting you to change some fact about the story. So there's an update at the bottom. Maybe you have to change the headline very rarely. I don't think we've even done that here at 404 Media. But that gives a reader the overall impression that the media is wrong in its portrayal of Facebook. And Facebook gets to have its own voice in a way that is criticizing the publication that is criticizing it. And I'm like really heated up about this at the moment because I just published a story today from Matthew Gault about Kanye's Nazi song all over Instagram. And there isn't a good story for Facebook to, to, to tell there. But they're trying to split hairs or find any way to push back on the article just for the sake of pushing back. And I think historically I'm like, I try. There's, there's obviously an antagonistic relationship between the press that is critical of whoever they are covering and the comms people at those companies or of those politicians or whatever. But I used to have the attitude that it's like, I have my job, they have their job. And a lot of the time they can be really helpful in providing information about maybe I did. I never want to be wrong in a story. So if I reach out to a press person and I'm like, hey, what about this thing happening at Facebook? And they're like, actually, you got this totally wrong. Let me tell you what is happening. That is very good. For me as a reporter to get that information, but increasingly I'm like, finding it hard to put myself in their shoes. It's like imagine you're the comms person at Facebook and Joe publishes this story about these students who are putting facial recognition in these glasses. And your remit from your boss is to go and chew his ass out or whatever you call it in the article, and you kind of like make your stand and fight with him. And like, I've had much so screaming matches on the phone with PR people and then, you know, a couple of months later they're like, fuck all that. It's going in the glasses anyway. It's just like, what the. How, like, what is their morale internally? And at what point you're like, this is so embarrassing for me as a human being to deliver these messages that I know are, if not lies, then bending the truth to the. To the point where it's unrecognizable.
[47:06]
Joseph
Yeah. And I think that's actually a really good place to end it all out. No, no.
[47:11]
Kate
I have one more thing to say. I'm so sorry.
[47:13]
Joseph
All I was going to add was that what you said, Emmanuel, where they pushed back and they're looking for an update with the student glasses. 1. They didn't even get an update in the end because we're not even going to entertain that because the points were stupid, you know. Sorry, Jason, go ahead.
[47:28]
Kate
Well, what I was going to say is often if you're doing a big story or a scoop story, a lot of the strategy is also like time delay too, where it's like, I'm going to get you on the phone, I'm going to ask you what your deadline is. I'm going to demand to talk to you off the record. We're going to talk and have like a. Maybe an interesting conversation off the record where the PR person is like, oh, I like, really want to help you. This is really bad. I'm going to have to go back to my boss and see what we can say on the record. Blah, blah, blah. Like, there's often a lot of that and you try to be nice to them on like a human level. Like a lot of it is like a human level. Like we, like this person says, I'm trying to figure out what's going on here. Please give me a few minutes to figure out what's going on here before you publish your story. So then they go and. And like say that they're going to do it and then half the time they either don't say anything at all, or they give you the same statement that they've given you a thousand other times. This happened a lot when I was writing about AI for, like, on Instagram, they give the same statement month after month after month. Or they, like, link you to a company blog post that they've already published. And in the meantime, maybe we get scooped. Like, maybe someone else publishes the story, maybe they take your story and they go to a friendlier outlet and say, Hey, 404 Media is going to write about this. Here's our side of the story. Why don't you write about it? And there's like a. We don't have time to get into it, but, like, some outlets are very friendly to companies, or sometimes the companies go out and put out their own blog post, preempting your story to try to get ahead of it and do damage control before you even publish it. And that dynamic is something that we need to consider every time we do a story. And it's really tricky. And it's like, it would. Whatever they're doing in, like, PR school, where they're just like, don't give out any information. I don't know. The relationship doesn't necessarily have to be super antagonistic, but. But we have to, like, draw a line somewhere where it's like, we've asked you for comment, we've told you the story is coming, we've given you a deadline. But, like, we can't always wait because half the time the companies will go. And, like, I'm not talking about meta, I'm talking, like, broadly. They'll, like, pull some fuckery that messes with our job and messes our livelihood. And that's something that we have to weigh as well.
[50:13]
Joseph
Yeah, and that's after you always are trying to be fair and reasonable. And then to some people it's more just of a game as well. Well, that was a really, really good group therapy session. I'm glad that we did that. How about we leave that there for the moment? If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about baseball and AI. I'm thinking of it as Moneyball 2.0. Jason couldn't correct me on that in a minute, although I think that's correct. It's going to be really interesting. You can subscribe and gain access to that content at 404 Media co. As a reminder, 404 Media is journalist founded and supported by subscribers. If you do wish to subscribe to 400404 Media and directly support our work, please go to 404 Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the Subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really does help us out. Here is one of those from MJ Red only podcast I pay for. Well worth it. Support independent journalism. This has been 404 Media. We will see you again next week.

Summary4 min read

The 404 Media Podcast: AI Avatar of Killed Man Testifies in Court

Release Date: May 14, 2025

Introduction

In this episode of The 404 Media Podcast, hosts Joseph, Sam, Emanuel, and Jason delve into a groundbreaking and controversial case where an AI-generated avatar of a deceased man, Christopher Pelke, testified in court during the sentencing of his killer, Gabriel Horquesitas. This unprecedented use of artificial intelligence in the legal system raises numerous ethical, legal, and technological questions.

AI Avatar Testimony

The episode opens with Jason introducing the core story: an AI recreation of Christopher Pelke is used to provide a victim impact statement during Horquesitas's sentencing hearing.

Jason [05:19]: "Horquesitas, the man who shot me. It is a shame we encountered each other that day in those circumstances, in another life we probably could have been friends. I believe in forgiveness and in God who forgives. I always have and I still do."

This AI-generated statement, which appears as if the victim is speaking from beyond the grave, is unprecedented in the American legal system.

Creation and Authenticity

The hosts explore how the AI avatar was created. Christopher Pelke's sister, Stacey Wales, authored the content of the statement, aiming to humanize her brother and express forgiveness, while his brother-in-law recreated his voice using AI tools.

Kate [07:41]: "Like, this was actually not the actual words were not generated by an LLM that was trained on Christopher Pelke or anything. Like she wrote his words or quote unquote his words and then her husband... used an LLM to train on his voice and recreate his voice."

The technology involved includes stable diffusion models and tools like Elora, which allow for the rapid training of AI systems to replicate a person's likeness and voice with minimal input data.

Legal Implications

Joseph raises critical legal questions about the admissibility and authenticity of such AI-generated testimonies.

Joseph [06:31]: "In almost a factual, cold, legal basis, it's not him talking. Right? I mean, what do you make of that?"

The discussion emphasizes that while victim impact statements are a standard part of sentencing hearings, the use of an AI avatar blurs the lines between genuine human testimony and technological fabrication.

Judge's Reaction

A significant moment in the episode is the judge's response to the AI testimony. The judge expressed appreciation for the AI statement, remarking:

Judge [12:26]: "I loved the AI and the family could have appreciated it, but so did the judge."

This acceptance by a legal authority surprised the hosts, leading to further scrutiny of judges' understanding and regulation of AI technologies in courtrooms.

Ethical Concerns

The hosts discuss the broader ethical ramifications of using AI to represent deceased individuals in legal settings. Kate points out the impossibility of truly knowing the victim's thoughts, highlighting the potential for misuse and emotional manipulation.

Kate [10:11]: "There is no way of knowing what he would have thought because... you can never know."

Emmanuel adds that society has a history of embracing new technologies in the context of death, often without sufficient critical thinking about the implications.

Emmanuel [08:36]: "There's a longing and like the unknown about death that people accept really wild ideas."

Technological Accessibility and Misuse

The episode underscores the accessibility of AI tools, making sophisticated recreations feasible for the average person. This democratization raises concerns about potential abuses, such as unauthorized use of someone's likeness or voice.

Emmanuel [08:36]: "People can put facial recognition tech into their smart glasses and dox strangers."

Family’s Intentions and Public Perception

Despite the family's intent to honor Christopher Pelke and elicit empathy, the use of AI has sparked debates about authenticity and respect for the deceased's true voice and intentions.

Kate [17:40]: "Our goal was to make the judge cry. Our goal was to bring Chris to life and to humanize him."

Conclusion

The episode concludes with the hosts reflecting on the intersection of AI, law, and ethics. They express concern over the lack of regulation and the potential for AI to disrupt traditional legal processes. The discussion underscores the need for comprehensive guidelines to govern the use of AI in sensitive areas like the courtroom to prevent misuse and preserve the integrity of legal proceedings.

Notable Quotes:

Jason [05:19]: "Horquesitas, the man who shot me. It is a shame we encountered each other that day in those circumstances, in another life we probably could have been friends. I believe in forgiveness and in God who forgives. I always have and I still do."
Kate [10:11]: "You can never know what someone would have thought... that's the really awful thing about death."
Emmanuel [08:36]: "People accept really wild ideas when new technology meets death because there's a longing and like the unknown about death."
Sam [15:38]: "You went to law school, bro. Like, what are you talking about? This is purely the conjecture of this guy's sister."

This episode provides a comprehensive exploration of a novel and unsettling application of AI in the legal system, prompting listeners to consider the profound implications of technology intersecting with human emotion and justice.

Loading summary

Transcript65 lines

[00:01]
Kate
The other words of the year, they were just as online, you know, Oxford's was brain rot, if you can believe it. Its first use was in 1854 by Henry David Thoreau.
[00:12]
Joseph
RIP Henry David Thoreau.
[00:14]
Sam
You would have hated TikTok.
[00:17]
Joseph
Wait, Candace, we should probably introduce ourselves.
[00:20]
Kate
You're so right, Kate. Hello, listeners. We are the hosts of icymi, or rather, in case you missed it, Slate's podcast about Internet culture where we're extremely.
[00:28]
Joseph
Online so you don't have to be follow ICYMI now, wherever you listen. Hello and welcome to the 404 Media podcast where we bring you unparalleled access to hidden worlds, both online and IRL. 404Media is a journalist, founder, company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content@404me co. I'm your host, Joseph, and with me are 404 Media co founders Sam Cole.
[01:09]
Sam
Hey.
[01:10]
Joseph
Emmanuel Mayberg.
[01:11]
Emmanuel
Hello.
[01:12]
Joseph
And Jason Kebler.
[01:14]
Kate
What's up?
[01:16]
Joseph
Let's go straight into one that you wrote, Jason, along with Matthew Golt, our regular contributor. The headline is, I loved that AI judge moved by AI generated avatar of man killed in road rage incident. Let me just straight away play some of the audio and video, if you're watching on YouTube, of this avatar. It's just a few seconds, I think.
[01:45]
Jason
Hello. Just to be clear, for everyone seeing this, I am a version of Chris Pelke recreated through AI that uses my picture and my voice profile. I was able to be digitally regenerated to share with you today. But here is insight into who I actually was in real life. Take a look.
[02:05]
Joseph
Then the video carries on and it shows actual footage of Chris with family members and friends. If I recall correctly, Jason, what did we just see in here?
[02:18]
Kate
Yeah, yeah. And then at the end, let's play one more little bit where he basically forgives the man who killed him, which is very odd or at least was very weird to watch. But yeah, this is Christopher Pelke. He's a man in Arizona who was killed in a road rage incident by a man named Gabriel Horquesitas. And what you're hearing is the end, the very, very, very, very end of the trial where Horquisitas is being sentenced. And so basically he had already been found guilty. And this is a sentencing hearing where as part of it, everyone is allowed to submit a statement more or less like a victim testimony, more or less. And so what this means is people who know Gabriel Horquacitas can you know, submit like testimony saying this was a one time incident, like he's a good family man, things like that, that like that. Those are sort of the other types of statements that were submitted. And then Christopher Pelke's family was like, you know, he was such an amazing person. We're very sad that he's dead. Of course, like this is just a normal part of sentencing. And then the judge considers all of that and comes up with how long the person should be in prison.
[03:48]
Joseph
So it happens every single day in cases all across the states. Very, very standard. And usually doesn't involve AI Usually doesn't.
[03:57]
Kate
Involve AI And I guess what I'm saying is like this AI was not cross examined or anything like that. Like this is a pre recorded statement and we'll get into how it was made. But the, this was submitted to the court as essentially Chris Pelke's quote unquote own statement. So in this case you had a man who was killed who from beyond the grave submitted a statement about who he was as a person and at the end like directly addresses the killer and says, you know, like I, I forgive you. Which is, it's just very wild to think about. I, I think that this is very much like, I think this trope is over way overused. But this is very Black mirror to me. Like this is the type of things that Black Mirror examined, especially early on in its run where it's like you'll be able to digitally recreate people after they die and they'll, you can like continue to have a relationship with them. In it was a man who was killed who is testifying at the trial for, I mean it ended up being a manslaughter, but testifying at the trial for the man who killed him.
[05:20]
Joseph
Yeah. And maybe here this is where we can just drop some more audio and video of that latter part of the video where as you say, he sort of directly addresses his killer to Gabriel.
[05:34]
Jason
Horquesitas, the man who shot me. It is a shame we encountered each other that day in those circumstances, in another life we probably could have been friends. I believe in forgiveness and in God who forgives. I always have and I still do.
[05:52]
Joseph
So how do we get. Well actually no, I was going to save this for later, but I almost just want to immediately address it. The family says this is almost like the victim impact statement from Christopher. That is not true. Right. It is a creation that's been made and we'll get into that. In a second by the family. And I'm of course not even talking about this family's grief or what they must be feeling. I just mean on almost a factual, cold, legal basis, it's not him talking. Right? I mean, what do you make of that?
[06:32]
Kate
Yeah, I mean, I think that this case is a little bit, you know, it's really weird because you have a grieving family and I don't want to downplay that at all, but you have a grieving family that's trying to figure out how to cope with this situation. And basically his sister, who we spoke to a woman named Stacey Wales, she wrote this. Like this was actually not. The actual words were not generated by an LLM that was trained on Christopher Pelkey or anything. Like she wrote his words or quote unquote his words and then her husband. So Christopher Pelke's brother in law is the one who recreated his voice. So they use an LLM to like train. Train on his voice and recreate his voice. And then they also used Elora, which Emmanuel has talked about a lot, is basically a mechanism for like recreating a, a person from an image. And so that is. That is what, like how this was made?
[07:42]
Joseph
Yeah, I mean, Emmanuel, could you maybe talk a little bit more about that tool just for a second? Like, is it an app? Is it a tool? Is it just free to use? Does it just require one image? What's the deal with it?
[07:54]
Emmanuel
I would really define it as a method most commonly used with stable diffusion, which is an open weights AI model that can do all sorts of things. People know it mostly from its ability to generate images. And essentially it's just a way of very quickly training stable diffusion to recreate the likeness of a person or an object or a scene or a style. It's a way to like training Stable diffusion took a lot of time and a lot of training. Data and allure is a way to use just a few images or inputs to make it do something very specific that it wouldn't be able to do otherwise.
[08:37]
Joseph
Gotcha. That makes sense. So very accessible. It sounds like, you know, basically anybody could use it. And I presume that's why, you know, somebody like a member of the public or whoever may turn to it. Jason, what did. Well, two things. Why did Christopher's sister want to make this? And then second of all, sort of what did they make of the result? Were they happy with it? Or so the why and how good was it?
[09:06]
Kate
Yeah, I mean, it sounds like she did this because she was having trouble writing her own victim impact statement for the trial. I mean, I sort of can't imagine being in this situation and knowing the words to say. And also, I mean, thankfully, I've never been in that situation, but sort of knowing like, what to even ask for or how to feel about it. It should be noted that the family actually asked for the maximum sentence time because the family was able to submit, like, what they wanted. But interestingly, like, Christopher Pelke's AI statement said that he forgives the person who killed him. And he didn't specifically. And I say he. But obviously this video doesn't specifically ask for anything. But the tone of it is one that's sort of like, it's very unfortunate that this happened. He says at one point, you know, in a different life, we could have been friends. And it's kind of hard to know, like, what to make of the whole thing, I guess.
[10:11]
Joseph
And like, truth wise, like, is that really what he would have thought? Like, what do you mean?
[10:16]
Kate
Well, I can talk more about that. I mean, I think that there's no way of knowing what he would have thought because it's just regardless of how well, you know someone, I think that, you know, I have lost someone who was very, very close to me. And the first thing that happened after she died was all of these people who barely knew her started saying, oh, she would have loved this. She would have said that she would have done this. And first of all, like, she wouldn't have done most of those things. Like, I know for a fact. But also you can never know. You can never know what someone. And that, that's one of the, like, really awful things about death. And I think that's actually, that's actually like really important to this case because this is fiction. It is fiction masquerading as a legal document. Because you have no way of knowing what Chris Pelke would have said or would have done because he is dead. And you just simply can't know that no matter how close you were to him. And so you essentially have like his sister who has written something for him and really, like, it's masquerading. I mean, people have called it a puppet. Like, this story went pretty viral. Not just our version, but it was widely covered in the press. And you just can't know.
[11:40]
Joseph
Yeah, because some other people covered it because the videos came out. And I'm gonna play another one in a second. And then we spoke to the sister and I got more detail. But you did mention, you know, kind of framing this as a legal document and it is literally like it's been entered into court. And this AI avatar has, in a legal sense, testified in this hearing, someone who should know about legal documents and the sorts that should be allowed and potentially shouldn't be allowed, you know, are judges. And here is the judge's reaction to seeing this AI played in their courtroom. I love that AI.
[12:27]
Kate
Thank you for that. And as angry as you are and justifiably angry as the family is, I heard the forgiveness, And I know Mr. Horticides appreciate it, but so did I.
[12:42]
Joseph
So the audio was not the best quality there, but basically the judge says that I loved the AI and the family could have appreciated it, but so did the judge. Some of the coverage didn't really focus on the judge's reaction. Of course, it's in the headline of ours because it's pretty important. What did you make of the judge's reaction when you saw it, Jason?
[13:08]
Kate
Yeah, I mean, I was really surprised that it was accepted into the court. And, you know, I'm not a lawyer, especially not a. Well, I'm not any kind of lawyer, but I have less familiarity with the criminal legal system than I do some of some of the, like, other types of lawsuits that we cover all the time regarding, like, lawsuits and privacy. Yeah, copyright. But I. I know very little about the proceedings of, like, a criminal court, but I have to imagine that judges have pretty wide leeway to determine what can be accepted and what cannot. So I was shocked personally that he accepted this statement and said that he liked it. And I think it really does raise questions about how well informed judges, lawyers, victims, suspects, et cetera, are about how AI tools work. And we know that, like, law firms are sort of foaming at the mouth to use AI tools because it can really cut down on the amount of time that it takes to do things, to go through, like, really big legal documents. Like, you know, Sam has written several articles about lawyers who have, like, written documents for the court that hallucinated, you know, other court cases because they used an LLM. And in some cases, like, the judges chastised the lawyers for doing so. And, you know, in this case, the judge didn't chastise the family for doing this. He said, thank you for doing this, and. And accepted it. I mean, I think Sam can talk a little bit more about how the legal profession is using AI But I don't know, I think it opens up a lot of questions, like, can you. Can. Can a victim's family call an AI chatbot trained on someone's text messages to you know, testify in a court, like against. In a murder case. Like, I have no idea. Like, it raises up. It raises all sorts of questions to me.
[15:23]
Joseph
Yeah. Sam, what do you think? On one side, we have the judge here embracing it, and then you've covered all of these cases where lawyers have basically effed up because they've been using AI in their own sort of writings. What do you make of it?
[15:38]
Sam
I mean, I thought that was a big part of why this was such a strange and kind of shocking story. A lot of judges are threatening lawyers with fines and fees and severe punishment for using LLMs in compiling documents for courts. Like, we see a couple times. Most recently, I think the MyPillow attorneys did this, but we've covered it a couple times where lawyers are compiling cases and they're citing other cases that are comparable. Like, you know, see this case, that case, to kind of hold up their own argument. And either the opposing side or the judge catches that the cases are just not real. They don't exist. That's because they're using AI. Who knows which one? There's many. But, for example, they're using ChatGPT. You can imagine them kind of typing in, make some cases, cite some cases for me as part of this case that I'm working on, and then they plug that straight into a court document and submit it. And that's really sloppy, and it would piss me off as someone who hired that lawyer. And it pisses off the judge because the judge is like, this is a waste of time. This is a mockery of the profession that you're in. So to see this judge in particular say, be so moved by it is just. I don't know. It's like, you went to law school, bro. Like, what are you talking about? This is purely the conjecture of this guy's sister. It's not anyone's words but hers and her husband's and the family's. So it's just. It was amazing to me that the judge was so. I don't know, I don't want to say gullible, but, like, that's kind of what it is. And then they were asking for the full sentence. So, like, really, was he so magnanimous as to forgive his killer?
[17:41]
Kate
It should be noted that the judge gave the killer the maximum sentence as the family asked for. So it's like, it's pretty. The. The forgiveness from the AI avatar did not sway the judge to go more lenient.
[17:57]
Sam
And it wasn't meant to.
[17:58]
Kate
It wasn't meant to because it came from the family. And the. The sister told us. Here's what she told us. She said, quote, our goal was to make the judge cry. Our goal was to bring Chris to life and to humanize him. And it's like, I think you can feel a lot of potentially different ways about it, because I'm sure that any family that's been through this will say that they feel like they've like, lost their voice. Like, Chris cannot speak for himself. He cannot speak to what happened that day. But that is just like how life is. It's like it's the definition of. Of a murder or, or any, like, any case where someone dies. It's like they cannot describe what happened to them.
[18:44]
Emmanuel
I just. Yeah, I want to note that there's a long history of people more likely to put their critical thinking aside where new technology meets death. And I think there. That's because there's such a. A longing and like the unknown about death that people accept really wild ideas. That's something that I covered a few times when we were at Motherboard. So, for example, in 2015, when VR was all the rage, there were quite a few projects that claimed to let you connect with your dead loved ones by creating some sort of VR experience that recreate the likeness of a lost parent or something and having some voice acting in there and that letting you communicate with the dead, which, again, obviously is not true. There have been a lot of kind of Holocaust memorial project that use holograms and other new technologies, which, again, are like, well intentioned, as I think the family is in this case. But as Jason said, it's off. You're not communicating with someone who has died. It is more like talking to a psychic, which I think as a culture we're more likely to acknowledge is a scam. But for some reason, when you put the label of technology on it, it is something that you can present seriously in a court. And with AI being the hot new technology, we've seen a ton of this stuff around death. I will remind people that Sam covered this company called Replica many times and that it is essentially an AI girlfriend app. It is a companionship AI model. But it grew out of this project where someone created an AI model of their dead partner and that kind of had this huge glowing feature in the Verge. And again, it was seriously engaging with this idea that this allowed someone to talk to the dead, and then five years down the line is just a sex robot of some kind of. So, yeah, just. Just like this is a thing that continues to happen over and over. Again, but it is definitely not something you'd see admissible in court, usually.
[21:27]
Kate
Yeah, I mean, you're absolutely right. And I would just add that one of the core projects of AI in general is to recreate consciousness. Like that is the, the goal of it. When we talk about, you know, super intelligence and the singularity and things like that, it's like to create a conscious being and we don't need to dive into it now. It's, there's many reasons to be very skeptical of this, but that's the entire selling point. That's like behind all the AI safety. And then there's a lot of, you know, transhumanists and like AI developers who specifically want themselves to be, to live on as an AI. They want their own consciousness to be like, uploaded into the everlasting cloud. And so I think that that is very often. There's so many AI startups that have this as their goal and just to sort of transcend death in some way. And there's been many, many projects like this. And so, yeah, to see it in a court, I mean, when I saw it, I was shocked. And it's like, I'm very rarely shocked by anything with AI anymore, but I'm like, oh, you've gotta be kidding me. Like, I cannot believe that this was accepted.
[22:46]
Joseph
Yeah, all right, maybe we'll leave that there. When we come back after the break, we're going to talk about our interactions with Meta and I suppose Silicon Valley PR reps more broadly and specifically, it's about Meta's plans to put AI, sorry, facial recognition, same sort of thing, into its smart Ray Bans. We'll be right back after this.
[23:26]
Kate
For years I've just been trimming my beard because I found shaving to be uncomfortable and razors to be expensive. Then I realized that I just didn't have the right tech. Staying well groomed requires the finest and most advanced tools. Upgrade with Harry's. Get the best razors money can buy for a fraction of the price of bigger brands. Right now you can get a $10 trial set for just $5 at Harry's.com 404, which is exclusive to our listeners. Like I said, I haven't regularly shaved my face for years because I have sensitive skin and didn't like dealing with the fuss. Then I got Harry's trial set, which comes with a premium five blade razor, foaming shave gel with aloe, which is great for sensitive skin, and a travel cover. I had the best shave I've had in years. Totally smooth and no razor bumps, which is huge for me. Harry's has the highest customer satisfaction in the shaving industry, a no risk trial and a convenient subscription model that you can cancel at any time. Don't like your shave? No worries, it's on them. Take your grooming out of the stone Age. Get Harry's Normally their trial set is ten dollars, but right now you can get it for just five dollars at Harry's.com404 that's our exclusive link Harry's.com404 for a five dollar trial set starting your own business is hot this summer, but lots of us don't vacation alone. And you shouldn't dive into starting a new business alone either. It goes a long way to have proven experts to help you navigate that great unknown. And sometimes it makes sense to have a tour guide. For millions of businesses, Shopify is their guide. A tool that simplifies and supercharges your business. Shopify is the commerce platform behind millions of businesses around the world and 10% of all E commerce in the US. It's also the tool we use for our own merch store. Setting up and managing our store couldn't have been easier thanks to Shopify's powerful, user friendly interface. Get the word out like you have a marketing team behind you. Easily create email and social media campaigns wherever your customers are scrolling or strolling and get started with your own design studio. With hundreds of ready to use templates, Shopify helps you build a beautiful online store to match your brand style. And best yet, Shopify is your commerce expert with world class expertise in everything from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into With Shopify on your side, sign up for your $1 per month trial and start selling today at shopify.com media. Go to shopify.com media shopify.com Summer is just around the corner and the folks at Mint Mobile have a hot take. Getting a summer bod is out and getting your savings BOD is in. This spring and summer we want skimpy wireless bills and fat wallets. And with premium wireless plans for just 15 bucks a month you can have both without breaking a sweat or the bang. Switching to Mint Mobile is a breeze. Soon you'll be getting great wireless coverage on vacation, at the beach or at the lake. Say bye bye to your overpriced wireless plan's jaw dropping monthly bills and unexpected overages. Mint Mobile is here to rescue you. All plans come with High speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of premium wireless service for Mint Mobile for 15 bucks a month. This year. Skip breaking a sweat and breaking the bank. Get your summer savings and shop premium Wireless plans@mintmobile.com 404 Media that's mintmobile.com 404 Media Upfront payment of $45 for 3 month 5 gigabyte plan required equivalent to $15 per month. New customer offer for first 3 months only. Then full price plan options available, taxes and fees extra. See Mint Mobile for details.
[27:51]
Sam
Well, well, well, we're back. Well, well, well, Joseph. What is going on here? So, Joe Rosery. The headline is well, well, well, Meta to add facial recognition to glasses after all. So if you're a longtime reader of 404, you know what we're alluding to already. But a while back, how long ago was it, Joe? Was it like maybe earlier this year or last year?
[28:25]
Joseph
I believe it was last October. October 2024.
[28:28]
Sam
Okay. So yeah, in October, these two Harvard students did something pretty wacky with Meta's. They have these Ray Ban glasses that you can wear and they have cameras in them. So these two Harvard students got a hold of these glasses and did something with them. So do you want to walk us through that original story first before we get into the smugness of it all?
[28:50]
Joseph
Yeah. So they take these Meta Ray Ban smart glasses. I always change the order of those words every time I say them. But they took those and they have the camera on board and then they basically strung together all of these different technologies which would include PIM Eyes, which is the off the shelf facial recognition tool basically anybody can use. People have used it to identify January 6th rioters. People have used it to dox sex workers. People have used it in all sorts of abusive and then somewhat interesting ways as well. But they paired basically PIM Eyes with these Ray Ban glasses and they called it Eye X Ray, if I'm remembering correctly. But it would also go a step further in that PIM Eyes gives you the webpage where that matching face was found online. So let's say you're looking at somebody with the glasses. It captures their face, it then puts their face through PIM Eyes and it brings up, let's say, the website of the law firm that this person works at. Then we could take the name attached to that, put that through an LLM and look for more information. And the idea was that it could basically dox somebody just by looking at them, finding where they live, finding where they work, all of this other stuff, and puts that all together into one package. And they made this video, which went viral on Twitter. X of the two students using it, they said, on dozens of people without their knowledge, like unsuspecting, and I would say victims, maybe some people were okay with it, but just going around and revealing the identities of people who were anonymous members of the public. And this is the big, big fear when it comes to putting not just facial recognition tech in the hands of anybody, but facial recognition tech in the hands of anybody in a surreptitious manner, in that I can just look at somebody and my Google Glass from a million years ago or now my meta ray bans are going to reveal to me who that person is. Where ordinarily they would be anonymous and they just walk down the street. I'm never going to see them again. Now I know who they are and I know everything about them. And I mean, it's a terrifying proof of concept. So we covered that at the time and we'll get into more detail in a second. But Meta pushed back a lot against that article and they didn't really like that we were reporting on it.
[31:34]
Sam
The wording that you used in the story that we're talking about today is they let me get it right. They chewed my ass off.
[31:43]
Joseph
Yes. That's a technical term.
[31:46]
Sam
Yeah. So with that context in mind, what is meta announcing now as of last week?
[31:55]
Joseph
Yeah. So there's a report in the Information that meta is actually working on facial recognition tech for its meta ray bans. And like, we don't know all that much, but basically they're working on it actively to put that capability into the glasses. And they're also doing, or they're also exploring this other thing, which is not having or potentially not having the light turn on on the glasses. So usually when you put the metal ray bans into recording mode, or whatever you want to call it, a little light will appear, which will indicate to other people, oh, hey, that person's recording. I don't think enough people wear these glasses for that to be common knowledge, to be perfectly honest. Like, you know, if someone's shoving an iPhone in your face or a GoPro in your face, you don't know the little red light on the pair of ray bans means you're being filmed. Regardless, Meta is reportedly looking at getting rid of that as well, so after all that coverage we did as students and have Meta's little meltdown about it. Well, well, well, they're putting facial recognition into the glasses themselves anyway.
[33:04]
Sam
Yeah. And they were super pissed when the first article with the students came out because obviously these are really, this is like a really privacy invasive thing to have people walking around doing facial recognition in real time to everybody around them without their consent. I don't know, I find that to be a pretty dark world to imagine living in. But Meta was like, you know, they were, I mean, walk us through what they said in response to your first story because they were pretty upset with like every element of the story. Yeah, like we said, they really did chew your ass off. I mean, they basically back and forth in emails.
[33:50]
Joseph
Yeah, they basically disagree with the premise. And I won't go into every single sentence they wrote. You can go and read the article for that, but I'll give like the summary of it when I reached out for comments saying, hey, look, these students have taken your technology made by Meta and Ray Bans, I suppose as well, and they've paired it with this facial recognition technology and these people search databases. Can I have your comment? Blah, blah, blah. Because we're writing this up now. And they initially asked, hey, just to clarify, this could be done with any sort of camera, Right. It's not something that's intrinsic or inherent to the Meta Ray bans. You could do this with a stationary camera or whatever, and then you've strung those different technologies together as well. I mean, yes, that is true, but as I explained to the Meta spokesperson, the students didn't strap a GoPro to their helmet or their face and then walk around trying to dox people, because that would be really, really obvious what they were doing. And again, by the student's own omission, the entire point was to do this in a stealthy manner and to do it on unsuspecting people. If you're shoving an iPhone in somebody's face, that's not going to be unsuspecting. We're all used to that by now. So they take issue with that. And I explained it. Whatever. They then take an issue with the headline, which we did, which was, someone put facial recognition tech onto Meta's smart glasses to instantly dox strangers. They had some sort of issue with the word onto Meta's smart glasses, where they're like, oh, you're suggesting that the glasses were modified in some way. And I'm like, I don't know, man. Onto into, slapped on, shoved together. It doesn't matter, man. It's completely meaningless distinction. And the impact is that this technology is being used in this way. We then record a podcast about it that people can go back and listen to if they want to get much more detail there. Headline of that was the smart glasses that docs strangers. And then the response was, and I think I'm just going to read this out. This is the only section I'm going to read out, but it kind of gives you insight into Silicon Valley press reps. And they said you say in the podcast that the glasses don't have facial recognition capabilities. And you've previously acknowledged that this could be done with any camera or recording device. But a headline saying smart glasses that dock strangers clearly makes it sound like this is the issue that is specific to the glasses, or that the facial recognition was executed on the glasses themselves versus the reality, which is that this was all run by a program on their laptop. This is despite the fact that students themselves have said publicly, we do not want this to be a criticism of their product at all. And we just had them on hand. This could have been done on a phone camera. And he finishes the quote and says, I realise we may not agree on everything here, but surely you can appreciate how headlines like this are misleading for readers, as I explained in this well, well, well piece. No, I don't appreciate that whatsoever. And I don't care if the Harvard students were saying, well, we didn't want this to be a criticism of Meta's glasses. It's like, dude, you took them and you built an instant doxing device. It doesn't matter what your intention was. And in fact, that's actually more worrying. You've clearly built it, not fully understanding what you've actually unleashed here. And this is just what it's like dealing with Silicon Valley reps. And I guess that's something I actually just wanted to ask you, Sam and Jason Emmanuel as well. Is this what it's like interacting with PR reps from tech companies? Like, what. What do you all think of that?
[37:41]
Kate
Is that a rhetorical question?
[37:42]
Joseph
Yes, yes, it's literal. Because I just. Because I want you to fill it in. As in, is this an extreme. This isn't like the most extreme thing. It's just a guy moaning at me, an email, so who really cares? It's because they now announced they are doing facial recognition. That's important. But, like, I mean, what did you make of it, Jason?
[38:03]
Kate
I mean, I thought it was absurd. I thought it was absurd then and now I think, well, well, Well, I don't know what else to say other than that. But yes, this is, it's interesting because I think we try to be as transparent as we can about how we do our journalism and what, what that means, like what, where our information comes from and that sort of thing. And I think part of us doing our articles is going to the people and companies that the article is about and saying, well, what do you think of this? Like, give us a comment. Like, here's a thing that's happening, like what's, what's going on? And we actually very often at this point have to go to companies and say, give us a comment. Like, answer these questions. Here's a statement, here's a comment. And that's actually not the ideal. The ideal is like make someone in your company available for an on the record interview to discuss this situation so we can have a back and forth dialogue. And that happens pretty much never anymore. I mean, I can't remember when was.
[39:14]
Emmanuel
The last time, when was the last time?
[39:15]
Kate
I mean, it happens never. And, but, and when it does happen, it's like Mark Zuckerberg is talking to a friendly podcaster for three hours. He's like talking to Lex Friedman for three hours. But like, when is the last time that anyone tech company did like an on the record, hard hitting interview with a real journalist? Like, I don't know when, I don't know when and when they do it. It's to promote like a specific thing. And so it would be interesting to have this debate with Mark Zuckerberg or you know, we understand he's a, he's a busy person. So how about someone who's working on the Smart Glasses project? How about an on the record conversation with the PR person? And it's like in this case none of this was on background, which means you can use the information or sometimes it means you can quote it but not say who, who it came from. Like the definition of what on background even means is different for every person. And if you're listening to this, you have no real reason for knowing what that is. But basically it's like when you deal with spokespeople at companies, they usually immediately demand to go off the record, which we can either refuse to do, which is tricky, tricky.
[40:34]
Joseph
Sometimes we do depends on the story, but sometimes we do it.
[40:36]
Kate
Yeah, sometimes we do it. I mean, I think we try not to because it's like not ideal, but sometimes we do just because we're trying to be fair to them. We're trying to explain, explain what the story is, we're trying to. Sometimes they want to go off the record because something is like a giant misunderstanding and it's like not actually a story and like, this is rare, but sometimes it's like we have something wrong and they want to tell us, like, hey, this is like wrong. And if they were to say that on the record, then maybe we would do a story saying like, I don't know, it's just like, it's a tough thing to even explain.
[41:13]
Joseph
Facebook denies XYZ when it's like, it's straight up wrong. But as you say, that's super rare anyway.
[41:19]
Kate
Yeah. And so it's like, it's like trying to get blood from a stone, I guess. It's like getting information from companies through official channels is like, they're doing pr. We are sort of doing like due diligence, just being like, yo, like, here's what we're writing. What do you have to say about it? And it's become like a really antagonistic type of back and forth or it's become like a box that you tick. Because company, like companies broadly, they're like, we don't give a shit. Like, here's, here's some like canned statement that doesn't mean anything. And then we put it in our article because we sort of have to, because that's just like how it works. And it's like half the time the companies are lying is the wrong word, but half the time they are like trying to like split hairs so finely that it doesn't mean anything to any reader. And as you said, Joseph, it's like onto into, taped to like, who gives a vibes? And then the other times it's like they are. They are doing some sort of like gymnastics with their boss where their boss is like very mad that a specific word or a specific company is in the headline or something and that. And they're like willing to go to the mat with you to like demand that you change a headline because someone at their company doesn't like it. And it's like, we don't change it unless it's wrong. And if it's wrong, then we'll fix it. But I don't know. That's a lot. But it's. Dealing with these people is a nightmare.
[43:02]
Emmanuel
It seems to me like the last time we had anything resembling a revealing conversation with a comms person at a tech company was back when Joe and Jason did this big feature about Facebook global moderation. And that was in the wake of Facebook being blamed for inciting violence in Myanmar and the genocide there. And back then, it seems like the comm strategy or their response to that was like, we're really sorry this happened. This was really bad. Here's what we're trying to do. Here's why this happened, despite our best efforts. And now we've revealed information that helps you better understand the company and we've learned lessons and yada, yada, yada, and that's the story. And I feel like since then, especially since Zuckerberg's transformation into like this right wing pro Trump figure is the PR people are just pushing back for the sake of pushing back on any story that they don't like because it's like a game and they want to score some points. So whatever it may be, they just want like you wrote a negative story. It seems to me my read on their strategy as a whole, and this is true about many tech companies, is that they want some sort of pushback in the article. And it almost doesn't matter what it is. It can be their comment, it can be a denial, it can be getting you to change some fact about the story. So there's an update at the bottom. Maybe you have to change the headline very rarely. I don't think we've even done that here at 404 Media. But that gives a reader the overall impression that the media is wrong in its portrayal of Facebook. And Facebook gets to have its own voice in a way that is criticizing the publication that is criticizing it. And I'm like really heated up about this at the moment because I just published a story today from Matthew Gault about Kanye's Nazi song all over Instagram. And there isn't a good story for Facebook to, to, to tell there. But they're trying to split hairs or find any way to push back on the article just for the sake of pushing back. And I think historically I'm like, I try. There's, there's obviously an antagonistic relationship between the press that is critical of whoever they are covering and the comms people at those companies or of those politicians or whatever. But I used to have the attitude that it's like, I have my job, they have their job. And a lot of the time they can be really helpful in providing information about maybe I did. I never want to be wrong in a story. So if I reach out to a press person and I'm like, hey, what about this thing happening at Facebook? And they're like, actually, you got this totally wrong. Let me tell you what is happening. That is very good. For me as a reporter to get that information, but increasingly I'm like, finding it hard to put myself in their shoes. It's like imagine you're the comms person at Facebook and Joe publishes this story about these students who are putting facial recognition in these glasses. And your remit from your boss is to go and chew his ass out or whatever you call it in the article, and you kind of like make your stand and fight with him. And like, I've had much so screaming matches on the phone with PR people and then, you know, a couple of months later they're like, fuck all that. It's going in the glasses anyway. It's just like, what the. How, like, what is their morale internally? And at what point you're like, this is so embarrassing for me as a human being to deliver these messages that I know are, if not lies, then bending the truth to the. To the point where it's unrecognizable.
[47:06]
Joseph
Yeah. And I think that's actually a really good place to end it all out. No, no.
[47:11]
Kate
I have one more thing to say. I'm so sorry.
[47:13]
Joseph
All I was going to add was that what you said, Emmanuel, where they pushed back and they're looking for an update with the student glasses. 1. They didn't even get an update in the end because we're not even going to entertain that because the points were stupid, you know. Sorry, Jason, go ahead.
[47:28]
Kate
Well, what I was going to say is often if you're doing a big story or a scoop story, a lot of the strategy is also like time delay too, where it's like, I'm going to get you on the phone, I'm going to ask you what your deadline is. I'm going to demand to talk to you off the record. We're going to talk and have like a. Maybe an interesting conversation off the record where the PR person is like, oh, I like, really want to help you. This is really bad. I'm going to have to go back to my boss and see what we can say on the record. Blah, blah, blah. Like, there's often a lot of that and you try to be nice to them on like a human level. Like a lot of it is like a human level. Like we, like this person says, I'm trying to figure out what's going on here. Please give me a few minutes to figure out what's going on here before you publish your story. So then they go and. And like say that they're going to do it and then half the time they either don't say anything at all, or they give you the same statement that they've given you a thousand other times. This happened a lot when I was writing about AI for, like, on Instagram, they give the same statement month after month after month. Or they, like, link you to a company blog post that they've already published. And in the meantime, maybe we get scooped. Like, maybe someone else publishes the story, maybe they take your story and they go to a friendlier outlet and say, Hey, 404 Media is going to write about this. Here's our side of the story. Why don't you write about it? And there's like a. We don't have time to get into it, but, like, some outlets are very friendly to companies, or sometimes the companies go out and put out their own blog post, preempting your story to try to get ahead of it and do damage control before you even publish it. And that dynamic is something that we need to consider every time we do a story. And it's really tricky. And it's like, it would. Whatever they're doing in, like, PR school, where they're just like, don't give out any information. I don't know. The relationship doesn't necessarily have to be super antagonistic, but. But we have to, like, draw a line somewhere where it's like, we've asked you for comment, we've told you the story is coming, we've given you a deadline. But, like, we can't always wait because half the time the companies will go. And, like, I'm not talking about meta, I'm talking, like, broadly. They'll, like, pull some fuckery that messes with our job and messes our livelihood. And that's something that we have to weigh as well.
[50:13]
Joseph
Yeah, and that's after you always are trying to be fair and reasonable. And then to some people it's more just of a game as well. Well, that was a really, really good group therapy session. I'm glad that we did that. How about we leave that there for the moment? If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about baseball and AI. I'm thinking of it as Moneyball 2.0. Jason couldn't correct me on that in a minute, although I think that's correct. It's going to be really interesting. You can subscribe and gain access to that content at 404 Media co. As a reminder, 404 Media is journalist founded and supported by subscribers. If you do wish to subscribe to 400404 Media and directly support our work, please go to 404 Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the Subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really does help us out. Here is one of those from MJ Red only podcast I pay for. Well worth it. Support independent journalism. This has been 404 Media. We will see you again next week.