Loading summary
Transcript53 lines
- [00:00]
Ryan Reynolds
Ryan Reynolds here from Mint Mobile with a message for everyone paying Big Wireless way too much. Please, for the love of everything good in this world, stop with Mint. You can get premium wireless for just $15 a month. Of course, if you enjoy overpaying, no judgments. But that's weird. Okay, one judgment anyway. Give it a try@mintmobile.com Switch upfront payment.
- [00:22]
Joseph
Of $45 for three month plan equivalent to $15 per month required intro rate first three months only, then full price plan options available, taxes and fees extra.
- [00:29]
Unknown
See Full terms@mintmobile.com are your ulcerative colitis symptoms proving difficult to manage? Tremphya Guselkumab can help you manage the cycle of UC symptoms. At one year, many patients taking Tremphya achieved clinical remission and some patients also achieved endoscopic remission. Individual results may vary. Tremphya is a prescription medicine used to treat adults with moderately to severely active ulcerative colitis. Serious allergic reactions and increased risk of infections may occur. Before treatment, your doctor should check you for infections and tb. Tell your doctor if you have an infection, flu like symptoms or if you need a vaccine. Ask your doctor if Tremphya can help you manage the cycle of UC symptoms. Call 1-800-526-7736 to learn more or visit tremphyaradio.com.
- [01:19]
Joseph
Hello and welcome to the 404 Media Podcast where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is journalist, founded company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404 Media co. I'm your host Joseph and with me are two of the other 404 Media co founders, the first being Sam Cole. Hello. And the other being Emmanuel Mayberg.
- [01:55]
Emmanuel Mayberg
Hello.
- [01:56]
Joseph
I think Jason's on a plane right now.
- [01:58]
Sam Cole
Maybe Jason's on a plane right now. I left Jason behind in Austin. We had a busy night. Last night was our our big takeover and after party Flipboard kindly hosted us for a shindig downtown for south by. So yeah, it was a. It was a long night. I'm a little. My voice is a little hoarse so I apologize. It's got a little fry more than usual and I'm a little hungover but we're here. Yeah, I just got in this morning so it was an amazing time. It was really good.
- [02:35]
Emmanuel Mayberg
Just as hungover by speaking loudly at a party than I do from the alcohol.
- [02:40]
Sam Cole
Yes, it's a whole. And yeah, I'm an inside cat. I do not go to networking events much unless we're throwing them. So it's, you know, it's like, it was amazing to see everybody and meet everybody and met a lot of podcast listeners, which was very cool. Yeah, it was a good time.
- [02:59]
Joseph
You heard that some people only knew about the event through this podcast, right?
- [03:05]
Sam Cole
Yeah, yeah, A few people came up and said, oh yeah, I heard about this happening on the pod. I was like, that's amazing. We plug it on the podcast because we know a lot of people listen. But it was cool to see how many people actually are tuning in.
- [03:21]
Joseph
Yeah. And it brings up one thing I'll just say super briefly, which is that clearly some people only listen to the podcast rather than reading the website. And at first when we sort of realized that, I found that quite strange. And then I realized, wait, I do that myself. I listen to the Verge podcast every single week, twice a week, if they've uploaded without fail. But then I generally don't read the website just because I prefer to digest it that way. So if you are one of those people who only listens to the pod, thank you very much. Really, really do appreciate it. And the pod is growing. With that said, let's get to our stories because we have a bunch of complicated and interesting stuff to get through. The first couple of stories which we're kind of putting together, they're written by Emmanuel and the headline of this first one, Chinese AI Video Generators Unleash a Flood of New non Consensual Porn. Emmanuel, this is something you've been working on for, for a long time. What is the top line of your investigation? Is it about the guardrails of these Chinese developed AI models? What's the top line?
- [04:26]
Emmanuel Mayberg
The top line of the investigation is that there are a bunch of AI video generators that are available via apps that you can get via your web browser or the app stores. And I don't think many people know about them because they come from smaller, lesser known AI companies. And to explain for a bit why I have been working on this for so long, I think we need to travel back in time to. I think it was February of last year when OpenAI revealed Sora, which is their AI video generator. And I think that's the first time that people saw kind of high quality AI generated video. And that really blew people away. It blew me away. They kind of came out with those video samples out of nowhere. And that tool wasn't available at the time and it wasn't available for a long time. It is available now if you pay.
- [05:31]
Joseph
They were more just showing it off, right?
- [05:32]
Emmanuel Mayberg
Yeah, they were just showing like, look at how powerful AI videos can be. And they definitely deliver that message. What I did that day is immediately go to the chat rooms where I monitor the communities that create and share non consensual AI generated pornography. And I wanted to see how they reacted. And obviously they were kind of salivating over having access to those kinds of tools. But it wasn't really a concern A Because OpenAI didn't give them access and they rolled it out very slowly and additionally very safely. OpenAI is notorious for having pretty strong, some would say, overbearing, unnecessary guardrails around all their AI tools. And that is true for Sora as well. But what happened in between the time that Sora was announced and today is that a bunch of other competitors in the A market rushed to launch competitors. And at first these competitors seemed not nearly as good as what we saw from Sora. But as everyone who's familiar with this beat by now knows, AI kind of develops at a very fast pace. And now there are a bunch of AI tools that produce pretty damn convincing video. And for reasons that we can speculate on here in a minute, they just have really, really, really bad AI prompt guardrails. Right. So people might remember one big story we did. God, it's 2023 now, I think. No, it was also 2024, but that was around people using Microsoft Designer and kind of writing prompts that tricked it to generate non consensual images of Taylor Swift. And that loophole we've seen replicated in many AI tools since then. But overall, the big AI companies realized that people were using this loophole and abusing it and have gotten a lot better at having guardrails against that sort of abuse. These newer AI companies, not so much. And over the months of seeing people find these AI tools, find the loopholes and just generating at this point mountains of non consensual videos of celebrities.
- [08:13]
Joseph
So yeah, it sounded like it almost started with the Taylor Swift stuff that we reported on, obviously. And you got, and I think Sam as well on that piece as well, got more information. And then everybody saw it because it went viral on, on Twitter or whatever. Now you're saying these other predominantly Chinese developed AI models are basically being used for the same thing, but to a much, much larger degree. It's not just some Taylor Swift stuff, it's all of these different celebrities. When it comes to the video generators themselves. What are they doing exactly? Do you give it a video and it autocompletes it? Do you give it a photo and it animates that? Do you give it a T text prompt? What does the user put in to then make all of this stuff?
- [09:03]
Emmanuel Mayberg
Yeah, so it's both. You can generate something out of what appears to be nothing, but it's not really nothing because it's pulling from huge data sets that the AI model was trained on. But you can basically write a text prompt and generate a video that way. Or, and I think this is important, you can do image to video, which you give it a still image and then in the text prompt you write how you want the AI tool to sort of animate that image. And the latter is harder to moderate because the text prompt, you can fairly easily filter out terms that you don't want people to use. And those can be names of celebrities, nicknames of celebrities, and a bunch of sexual terms. Right. That's like a fairly easy way to filter out a bunch of bad content. You could do that with images, but that is much more complicated because then you need to train other AI models to recognize a person in the image or recognize nudity in an image. And that just takes a lot more effort to filter out those kind of visual prompts. It just appears that the AI tools that I've found, most of which are developed by Chinese companies, are not doing that very hard work of visually detecting images that are used to animate pornography.
- [10:42]
Joseph
Yeah, because it could be a red carpet photo of a celebrity or something like that. It's like, what, you're going to ban all red carpet photos? Well then they'll just find another photo or something like that. Right, so what are people making exactly? Is it like sort of lewd images where maybe they get a celebrity flashing their breast or something like that? Or is it, um, you know, more full on pornography? Obviously there's degrees here. I mean, it's all non consensual and it all sucks. But like, what are people making broadly?
- [11:17]
Emmanuel Mayberg
So I think if I went into these chat rooms and put all these videos into a spreadsheet and counted what is the most popular type of video? I would say it is probably videos of female celebrities taking their tops off. The reason for that, I think, is that there's a very popular tool called Pixverse, which is just, I think to be fair, is used for non harmful reasons by a lot of users, but it's just an easy accessible tool. You can get it on the web, you can get it via the Apple App Store. So it's very easy to access. And this community has figured out how to abuse it in this specific way. They found the specific written prompt that you can use to create that kind of video, and it's really easy. So that is the most common one. But what I saw is that people move from tool to tool depending on what kind of video they want to generate and what kind of vulnerabilities they're finding in each tool. So Pixverse is good for that. Then there's this other tool I talk about later, which is actually from an American company called Pica. And that one can, I mean, produce straight up like videos of oral sex. Fairly easy. And it looks a bit janky, definitely looks weird. But it is also, I think would be fairly horrific to find someone doing this to your likeness.
- [12:57]
Joseph
Yeah, and again, it just takes an image or something like that, depending on the tool. But it's very low effort if they have the prompt workaround. What specific Chinese tools are we talking about and are they like little upstarts? Are they like open source projects? Are they well funded operations? Who are these or what are these tools and sort of where do they come from?
- [13:26]
Emmanuel Mayberg
It's this new crop of AI tools that are doing this exact thing I think I talked about in the beginning, which is OpenAI presents Sora. And everybody sees the potential in that. And rather than be careful and release it very slowly and safely, their tactical move is to just get to the market first, get something that is honestly probably not as powerful as Sora and definitely not as safe, but it's really easy to access and there is a demand for this kind of tool. And just getting there first and giving people access, that's just a better business strategy or the only competitive business strategy that they have. They are well funded, they all have millions of dollars in venture capital. For example, this app that I talked about, Pixverse, it has some notable people. Like the person at the head of that company used to be the head of machine vision at ByteDance, which is the company that owns TikTok. So they're new companies. It's like a new generation of AI companies. We're seeing similar type and scale of company in the US but in this case they happen to be almost exclusively Chinese.
- [14:57]
Joseph
Yeah, and there's a line in the piece where you say 404 Media is not sharing the exact prompts that produce these videos. Can you explain why? I mean, I think it's obvious, but I think it's Useful for people to hear why you don't include these exact prompts. But then also, can you just mention. So you won't mention the prompts, but you'll mention the companies. Like, is that because, well, they're massive, multimillion dollar companies. Like, why wouldn't we name them? What's the thinking there?
- [15:28]
Emmanuel Mayberg
Yeah, I mean, as Sam knows very well, when covering this kind of thing, you're always trying to walk this very complicated line where you want to report about an important issue, you want to name and shame basically these companies and hopefully apply pressure not just on them to build better protections, but also build pressure on Apple and Google who are making these apps accessible via the App Store. But at the same time, we obviously don't want to teach people how to create very harmful content. So we're kind of sharing the responsible parties here, but we're not sharing the communities where people will teach you how to do this or the specific prompts that will generate those harmful videos.
- [16:22]
Joseph
Yeah. Sam, what do you think of that balance between highlighting something because there's a public interest is important while not amplifying the bad stuff? Like, how do you figure that out in your head?
- [16:35]
Sam Cole
Yeah, I mean, it's just so hard to write about something and especially to illustrate what's going on without saying what's going on. Like saying it plainly. I think when we kind of beat around the bush, so to speak, and try to use like euphemisms or descriptors that aren't exactly not the prompts, but, you know, like naming what these companies are saying. You know, for example, like last week we talked about Instagram Gore. It's like saying exactly what people are seeing without being cute about it I think is really important as journalists. So it's, it's a hard calculus. I mean, it's definitely something that I struggled with a lot early on, kind of figuring out when to be very blunt about these things and name the companies and when to not. And I think a lot of it comes down to it's, I think just, I think a lot of people's reaction when they see like, oh, you're talking about like a telegram group or an AI model or a tool or whatever it is. It's like they're like, oh, I haven't heard of that. So you're amplifying it. It's like, no, actually tens of thousands of people have heard of this, you just haven't. You know, lots of people are using this and this company's making a lot of money on those people. And it's doing, in a lot of cases, real harm to a lot of people. Just because you haven't heard about it doesn't mean it's not already a huge thing. Maybe it'll become more of a mainstream thing and more pressure will be put on it to, like Emmanuel said, be taken off the App Store and things like that. But that's also out of our hands. It's not really part of our job to do that. So, yeah, it's a tricky thing, for sure. It's something we think about, I think, every time one of these stories comes up.
- [18:20]
Joseph
Yeah. I mean, I think the stakes of this example, I'll say, are lower. But almost the quintessential one, I always remember is when Gorka first covered the Silk Road website, people were debating like, oh, you shouldn't do that, because now people will know to go do it. And it's like, I don't know, man. The revolution in marrying Bitcoin and the Tor anonymity network to allow the borderless online exchange of narcotics is probably something that's worth putting in an article, and it applies here when it comes to the unbridled use of this AI technology and on that. So just to get back to the models we're talking about a little bit. Emmanuel. So it's mostly Chinese in this article. There are some US Ones, or one you did mention. Is it more that the new generation of ones without guardrails just happen to be Chinese and that's sort of why Chinese is in the headline? Or is that sort of a commonality when it comes with Chinese companies that they just don't have these guardrails? How exactly does the China element play into it? Because, of course, you're always careful, and we all are here, but recently we had the deep seat stuff, and people lost their fucking minds over the Chinese element. That's a little bit different because it's like, well, you're giving data to a Chinese company, blah, blah, blah. But is it more they just happen to be Chinese here or what's that?
- [19:48]
Emmanuel Mayberg
I think there's two things that are happening. There are American companies that are doing this. There is American competition. But I think there is less. We've written about some of them. I wrote a story about one app called Dream Machine. Sam wrote a huge scoop about. I forget the name. The AI video generator that we got the training data on.
- [20:18]
Joseph
Runway.
- [20:19]
Emmanuel Mayberg
Runway.
- [20:19]
Joseph
Runway.
- [20:20]
Emmanuel Mayberg
Yes. Thank you, Runway.
- [20:22]
Joseph
We're running all our stories.
- [20:24]
Sam Cole
That's a long time ago. A lot has happened.
- [20:26]
Emmanuel Mayberg
Yeah, so they exist, but I think one thing that is definitely happening is that just that the Chinese companies saw an opening. We're like in this great competition in the AI industry between the US or the west in China and it was just a place for them that they can get ahead a little bit. And they did. And I think that's one thing that is happening. The other thing that I think is happening, and I haven't been able to prove this and I didn't put this in the article, but I feel comfortable saying it here and I invite people who are listening who might be interested in safety or red teaming and might be able to teach me about this, honestly. But I do suspect or I wonder if there's a language barrier problem here where the American companies are just better at building what we call like the semantic filtering. Right? Like the word based filtering of prompts where the Chinese companies, since they're initially built for Chinese markets and are prompted in Chinese, maybe have like fairly good filtering in Chinese, but the English language filtering is not as good. I was wondering if that's one issue here, but I don't know that for a fact.
- [21:42]
Joseph
That's super interesting. Yeah. I think there's one more thing you wanted to mention on this story, Emmanuel, before we move to Ali Baba. There was the apps and the models, Right?
- [21:55]
Emmanuel Mayberg
Right. So just to transition here to the next story, so far we've been talking about apps. These are user friendly, consumer grade, anyone can use them, are advertised to the average user, type of tools. The other thing that is happening at the same time is kind of a rerun of what we've seen with this website called Civitai and these more open models as opposed to apps. So there's two Chinese companies, Tencent and Alibaba, which are like two of the the biggest tech companies in the world. And they have released essentially the video version of Stable Diffusion, which Stable Diffusion also does video, but Stable Diffusion is basically an open weights model. It's a AI image generation model that you can tinker with to customize it and make it better at producing specific type of images. And Tencent released this tool called Hanyun, I hope that's how you pronounce it. And Alibaba produced this other model called Wan, just the exact same thing. They released all the documentation. There is a GitHub where you can go and download the code and tinker with it. As soon as this happened very rapidly, the exact same thing we saw with AI images happened. The models were adopted by the Civitai community. They were modified to create videos of Highly specific sexual acts and fetishes, and then also videos of very specific small time YouTubers and Twitch streamers, Instagram influencers. And while Civitai at this point is pretty good at preventing you from posting non consensual content to its website, it also makes it incredibly easy to like, I'm gonna take this AI video model that has been designed to create videos of blowjobs, and I'm gonna take this other AI video model that's been designed to recreate the likeness of this Twitch streamer that I like. And you kind of put them together and make non consensual videos which are also of much higher quality than what these apps that I talked about do. But at this point they are more difficult to produce. You need to navigate Civitai, know how to run these models. Either do it locally on a fairly powerful GPU or rent that GPU time in the cloud and setting up the workflow for that. And it's not impossible. Like, I could figure out how to do it, Anyone can figure out how to do it, but it is several degrees more difficult than just downloading an app and clicking Generate.
- [25:02]
Joseph
Yeah. And I think just the last thing on that before we take a break is when Alibaba released this open video model and then obviously then got used for porn, as you reported. What was the actual intention with releasing this? Why did they want to release it, and what were they hoping it was going to be used for?
- [25:23]
Emmanuel Mayberg
That's a complicated question to answer because it gets into this greater debate of like, why is Mark Zuckerberg releasing Llama as an open model? Right. So the theory is that it becomes widely adopted across the world and then question mark, question mark, question mark, monetize it somehow. It's how that works out that's kind of like above my pay grade. But you're just like, the plan is to make it open so as many people as possible adopt it. So the technology is developed by a community along with the company and has a lot of investment from that community. And then you sell them something. I don't know how that last stage works out, but it's the open model of AI.
- [26:15]
Joseph
Step four, profit. You don't need the in between steps. That's just how it works. All right, we'll leave that there. Really, really amazing stuff. When we come back, we're going to be talking more about AI. This entire episode's about AI. I mean, we're going back to almost our 2023 roots, but it's going to be about how police are using AI when it comes to analyzing seized evidence. We'll be right back after this.
- [26:58]
Unknown
I don't know about you, but I like keeping my money where I can see it. Unfortunately, traditional big wireless carriers also seem to like my money too. After years of overpaying for wireless, I finally got fed up from crazy high wireless bills, bogus fees, and quote unquote free perks that actually cost more in the long run and switched to Mint Mobile. Switch to Mint Mobile and you'll save in the short and the long run. It's easy to switch. You can keep your device, your phone number and your money. Say bye bye to your overpriced wireless plan's jaw dropping monthly bills and unexpected overages. Mint Mobile is here to rescue you with premium wireless plans starting at 15 bucks a month. All plans come with high speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of premium wireless service from Mint Mobile for 15 bucks a month. If you like your money, Mint Mobile is for you. Shop plans@mintmobile.com 404 Media that's mintmobile.com 404 Media upfront payment of $45 for 3 month 5 gigabyte plan required equivalent to 15 bucks a month new customer offer for first 3 months only, then full price plan options available, taxes and fees extra. See Mint Mobile for details. We're journalists, not business people. At least we weren't until we started 404 Media. Starting the business meant figuring out how to open up an online merch shop, which is something that I was honestly dreading. Then I found Shopify. Shopify is the all in one online and IRL store solution that you can get set up in minutes. Nobody does selling better than Shopify. Home of the number one checkout on the planet. Shopify's not so secret. Secret, it's Shop Pay, which boosts conversions up to 50%, meaning way less carts going abandoned and way more sales going. If you're into growing your business, your commerce platform platform better be ready to sell wherever your customers are scrolling or strolling on the web, in your store, in their feed, and everywhere in between. Upgrade your business and get the same checkout we use with Shopify. Sign up for your $1 per month trial period at shopify.com media. All lowercase go to shopify.com media to upgrade your selling today. Shopify.com media.
- [29:29]
This episode is sponsored by BetterHelp. Therapy can feel like a big financial investment with a single session costing anywhere between $100 to $250. For most people, that can be out of budget. But with better help, you can save on average 50% per session. And more than anything, it's an investment into your mental health. And your mental health is worth it. With therapy, you can learn positive coping mechanisms and learn how to set boundaries with yourself and others. Therapy isn't just for those who have experienced major trauma. It's also for anyone who wants help with navigating their everyday life experiences just a little bit better. With over 30,000 therapists, BetterHelp is the largest online therapy platform. The platform has served over 5 million people around the world. And with BetterHelp, you can switch therapists at any time. And if you're busy like we all are, you can join a session with just a click of a button. Your well being is worth it. Visit betterhelp.com 404 Media today to get 10% off your first month. That's BetterHelp. H E L p.com 404 Media.
- [30:46]
Emmanuel Mayberg
Okay, so this next story is from Joe. The headline is Celebrate is using AI to summarize chat logs and audio from seized mobile phones. I think let's start with what is Cellebrite? A notorious company in our little world, but for people who don't know, what is Cellebrite?
- [31:05]
Joseph
Yeah, so Celebrite is an Israeli company. I mean, it has US subsidiaries and stuff, I'm sure, but predominantly an Israeli company. And it's basically ubiquitous in the world of law enforcement. So when a police officer seizes a mobile phone, maybe that's at the border with cbp, maybe that's ice. When they arrest somebody, maybe it's a cop at a traffic stop. What they'll often use is a piece of technology from this company called Celebrate. And you know, it comes in lots of different forms, but generally the tool is called, I pronounce it U Fed. I'm not sure if that's entirely correct, but I like that it has Fed in there, ufo. And you plug the phone in, if it needs to, it will crack or bypass the password or the passcode requirement. And then it will download all of the data on the phone. You know, in some cases able to get deleted chats or chats that you thought were deleted, but they weren't forensically deleted necessarily. And then it takes all of that information and the police officer can, you know, safely store it for later, you know, if there's, I don't know, a murder investigation and you have the phone of the victim, you want a forensically sound image of that phone. Or if someone's crossing a border, you just want to download everything on their phone and rummage through it without a warrant, because that's what authorities are able to do. There's a funny sidebar about how Celebrate started, which is that when you would go into the Apple Store and you were like an Android user, but you wanted to change to Apple iOS, Apple would have Cellebrite devices in there because it could help transfer your data. I don't think that's the case anymore because I don't think that's necessary. But it's just funny, especially around the San Bernardino attacks of 2016, and there was all of the FBI versus Apple stuff, there's a lot of coverage about Celebrate because people were speculating that Celebrate was the company that unlocked the phone for the FBI, but I think it was. Kim Zeta at the Intercept actually did a really good profile of Celebrate and how, yeah, you go to an Apple Store, there's actually a Cellebrate in the back, probably. But, yeah, it's a very, very common tool and company across law enforcement. Its main competitor being Grey Key, which has probably taken actually a sizable chunk of its customer base.
- [33:35]
Emmanuel Mayberg
I imagine every once in a while, someone on Reddit posts a picture of the cellebrite device at the Apple Store, and it's like the Leo DiCaprio meme pointing like, oh, there it is. So Celebrite is doing what every other company in the world is basically doing right now, and they slapped AI on it, which is what your story is about. What does that mean for them? What does it mean in this context for them to use AI?
- [34:02]
Joseph
Yeah, so they've slapped AI into their products called Guardian. Again, that's not the one that's actually getting the data from the phones, but after the cop has extracted the data, they upload it to this system, like an evidence sharing system. This is a bad analogy, but it's almost like Google Docs for cops, where you can collaborate across the cloud, live with each other on a piece of evidence. It's like that, but for stuff that's been taken from mobile phones. So you'll have all of the chat logs in there, the voicemail memos, the photos, all of that sort of thing. And what Guardian is now capable of doing with this slapped on AI, it can summarize all of that material. So rather than a police officer having to read through every single text message or listen to every single voice memo. Guardian can use AI to potentially summarize it. Now I haven't seen sort of a, a Guardian produced AI report and I would absolutely love to see one if anyone gets hold of one. But that's the way they frame it in that it can really speed up investigations, it can save cops time, it can do all of the promises of generative AI essentially. But this isn't a school kid trying to generate an essay. This is a copy generating a summary of evidence seized from a mobile phone. And that is very, very different in my opinion. You know, the stakes here are a lot, lot higher when it comes to sort of this use of AI.
- [35:42]
Emmanuel Mayberg
Do we know what cops think about this new feature?
- [35:46]
Joseph
Yeah. So there's a couple of testimonials included with Celebrate's announcement. And I should say, like this actually came from a press release from Celebrate in February. I'm not really in the habit of reporting on press releases, but it seemed like nobody picked this up. I was searching around for something probably to do a Celebrate because of all these leaked documents we get from Grey Key and Celebrate, that sort of thing. And then I came across this February press release saying, yeah, we're putting AI into this product. So included in there are testimonials from police officers. Now this is very, very common. We saw it with Ring when we did a ton of reporting back at Vice where the cops would provide, you know, pretty positive reviews. And then, I don't know, they get cameras or whatever in exchange. Right. I've seen that with other surveillance companies and with this mobile forensics firm as well. So there is a testimonial in there and it comes from a pretty small police department who says they were piloting the AI capabilities. And they said, quote, it is impossible to calculate the hours it would have taken to link a series of Porch package thefts to an international organized crime ring. The Genii capabilities capabilities within Guardian helped us translate and summarize the chats between suspects, which gave us immediate insights into the large criminal network we were dealing with, end quote. So there's a couple of things. There's the first is that translating. So, you know, I mean, that's not that revolutionary, obviously. And I don't have a super problem with cops using an automatic translation tool, as long as it's verified later on. But then summarize the chats between suspects now again clearly into Genai territory. But also they say they've linked these different cases together by summarizing the evidence. Now, of course, maybe if not, most likely they would have linked those together as well, manually. I don't know, they're reading the chats or something and the same money mule comes up, or the same safe house comes up across all of these different chats, whatever. So maybe they would have found out about it anyway. But they're saying explicitly here that they linked all of these together with this AI tool? Basically, yeah.
- [38:07]
Emmanuel Mayberg
I think one of the reasons it is so easy for us to write articles about AI and make fun of AI is that one of the most common uses for it is summarizing other large pools of information. And it so often gets it wrong and it just increases in severity or like, becomes less funny, depending on the context. Right. So if it's a Reddit search, I mean, a Google search, and it's pulling some random comment from Reddit that tells people to put glue on their pizza, that's not great. But mostly it's like, ha, ha, AI is stupid. And then increasingly we've been seeing more and more stories about lawyers who are using AI in court. And then you get the AI hallucinating citations of cases that never took place. That seems pretty bad. And I sort of believe the cop quotes here about it being much faster to summarize just like massive amounts of information, like, sure, it's easier, but it's like, yikes. Yeah, and you got some pretty alarming quotes from civil liberties experts to like speaking to this exact issue. And what, what do they say?
- [39:29]
Joseph
Yeah, and this is from Jennifer Granig from the aclu, and, you know, she brought up Fourth Amendment issues, which is like, when you get a warrant, it's only supposed to be for a particular device and maybe even a subset of that device and that sort of thing. And that's obviously a long standing Fourth Amendment warrant issue. But I think she brought up some other really, really interesting points, which were, you know, she said there could be a tendency to believe that an AI tool will successfully identify patterns which reveal criminal behavior more so or better than the human. Reviewer. So you could end up trusting the AI more because, oh, well, this is an AI tool sold by a law enforcement contractor. Like, why would I not trust it? You know, that said, and I think we'll get a little bit into this at the end, but Celebrate says there is always a human in the loop. An hi, I think they made that acronym up because I've never heard that. And when Jason was editing it, he highlighted it like, oh my God. So there's always a human in the loop. They say, okay, but I Don't know. This is still crazy to me. And the idea that it could introduce errors which then the police officer has to catch and maybe they will catch. But wouldn't it be better if they just did it from their own experience? I'm not entirely sure. But you brought up the one how it's easy to dunk on chatbots, and here the stakes are higher. I mean, we're seeing more and more of this. There was this BBC study. When was this from? From February. And they tested, it seems, quite methodically, ChatGPT, Copilot, Gemini and perplexity to summarize news stories. So basically, it's kind of doing what's the same here, summarizing some sort of corpus, some sort of body of work, and found that 51% of all AI answers to questions about the news had significant issues of some form. Half of them are getting the shit wrong. That is really, really crazy. When you're talking about, obviously, news that's very, very bad. But then in this context as well, I don't know, it's just alarming and concerning, for sure. And again, I haven't seen any errors in it, but I think the potential is absolutely there.
- [41:52]
Emmanuel Mayberg
This is not going to be the end of AI and police work. There are several stories that we're working on that we'll dig into this more in the future. But what do you think about investigators? I don't know, the FBI, any law enforcement agency increasingly adopting AI tools into their investigations, their reporting of crimes.
- [42:21]
Joseph
Yeah, so there's a couple of examples. One is the thing I go on about most in the entire world, which is how cops are increasingly compromising entire encrypted chat platforms like sky, encroachat, Anom, and then all of these other smaller ones exclude Ghost Matrix, blah, blah, blah. The Dutch authorities in particular, they did create AI tools to surface content in those massive, massive data sets. So if a criminal was talking to another one about cocaine, for example, the AI would surface that chat, tell an analyst, hey, here is a conversation about cocaine. And in my reporting, speaking to those law enforcement officials involved in those sorts of investigations, it seemed that that sort of AI was especially useful. I guess the good thing there is that it's very much limited to that data set. But then it still brings up questions of, well, eventually those people are going to be prosecuted. So it would still involve the human going and verifying the actual evidence against them. That's one way. The other one, which I think is probably a lot more relevant for more people, is that AXON the law enforcement contracting giant, you know, it makes what, tasers, body cameras, basically everything for everybody, right? And they have this relatively new capability called Draft One. Yeah, Draft One. And it uses basically ChatGPT or OpenAI's model to summarize the audio of body cam footage. So it takes that audio, it listens to it and basically summarizes what happened. And examples are, oh, a man came over to the police officer and he said xyz, he described the suspect as blah, blah, blah. And the idea is that police officers can be sort of more engaged in the moment. They can be talking to the witness or the victim or the suspect, and they can be really engaged in that conversation without having to like sort of remember bits and bobs and that sort of thing. And when I was going through a lot of the Axon material, they were saying, or maybe the police officer was saying that, that with the rise of body cameras, officers have now they speak more to the body camera than to the person in front of them because they know they're being recorded. So they say they almost repeat everything. So it gets recorded on the body camera in a way. I don't know. That sounds like a good thing. There's more evidence, right? But the idea of that AI is that the cop can just go and sort of do their job and then Axon's tool, Draft one will summarize it. I mean, that made quite a splash in good and bad ways when it was announced several months ago at this point. And it brought up basically the same concerns as the celebrate one here, which is that you're asking an AI to summarize stuff, which is really, really important. This is not trivial, this is not some dumbass lawyer. And to be clear, the stakes are pretty high there as well, but the judge is always going to catch them here. It's a lot more asymmetrical because, well, it's the cops generating the evidence and summarising it with their AI. You don't know if the victim or the witness or the suspect or anything necessarily gets a chance to challenge that in the moment. Right. It's a black box happening over there. So I don't know. AI is going to continue to become a more and more relevant part of policing. And I think we're going to start to see the side effects of that in the same way that we saw side effects with facial recognition where more cops were using it. And yes, it's an exceptionally powerful tool for them. But the wrong people have been arrested because they happened to be black or something like that and these systems make mistakes. Does that sound. Does that sound right? Emmanuel?
- [46:33]
Emmanuel Mayberg
That sounds horrible. So yes, it sounds right. It sounds extremely frightening and bad. So yeah, sounds correct.
- [46:39]
Joseph
That is what we do on this podcast. If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about the rise of vibe coding and a game someone made with AI that's now making them allegedly $50,000. Which is crazy. You can subscribe and gain access to that content at 404 Media co. As a reminder, 404 Media is journalist founded and supported by subscribers. If you do wish to subscribe to 404 Media and directly support our work, please go to 404 Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You also get to listen to the Subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really, really does help out. This has been 404 Media. We'll see you again next.