← The 404 Media Podcast
Loading summary
Transcript92 lines
- [00:00]
Jason Kebler
Foreign.
- [00:05]
Joseph
Welcome to the 404 Media podcast where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist family company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content@404Media co. I'm your host, Joseph, and with me are 404Media co founders Sam Cole.
- [00:35]
Sam Cole
Hello.
- [00:35]
Joseph
And Jason Kebler.
- [00:37]
Jason Kebler
What's up?
- [00:39]
Joseph
As we're talking, I'm getting notifications that Jason is telling Emmanuel I don't know how to zoom in on my new camera, but maybe we'll save that for another time.
- [00:49]
Jason Kebler
Yeah.
- [00:49]
Joseph
Okay.
- [00:50]
Jason Kebler
Joe's been setting up his camera for like three weeks now.
- [00:53]
Joseph
I do it bit by bit, as in I'm slowly improving the podcast set up.
- [00:59]
Jason Kebler
He's reading the instruction manual on how to turn camera on.
- [01:02]
Joseph
Well, we got that far.
- [01:03]
Sam Cole
It looks really good.
- [01:04]
Jason Kebler
This is not it. This is not it. This isn't the camera. This is not it.
- [01:08]
Joseph
No.
- [01:08]
Sam Cole
What are you talking about?
- [01:09]
Joseph
The audio's new. This is just my standard webcam.
- [01:11]
Sam Cole
You're just looking sharp today.
- [01:12]
Jason Kebler
Like he was showing me the inside of his nose moments ago.
- [01:16]
Sam Cole
Oh, well, I missed that.
- [01:18]
Joseph
Well, that's the pivot.
- [01:19]
Sam Cole
We're pivoting subscriber's content instead of Joe's nose.
- [01:23]
Joseph
Exactly. One bit of housekeeping before we get to the stories. We have a reader listener survey. There isn't a fancy link like a bitly, but I will put a link in the show notes. Jason, do you just want to explain why we're doing this and sort of why it's important?
- [01:43]
Jason Kebler
Yeah. We are on Ghost, which we've discussed before, but is the architecture that our website runs on and it has no cookies, which is very good. It's very privacy focused, privacy centric, which is important to us because we care about these sorts of things. And, you know, we don't want to track our readers all over the Internet. That said, we don't know very much, like really anything about the people who subscribe to us and what things you like, what things you don't like. So this is going to help us as we decide sort of what to do, which projects to prioritize. Going into 2026, we have questions about, like, do you like our email newsletter? Do you want it to be different? Do you want to see us cover different things? Would you be interested in a print product like A magazine, things like this. And then, I mean, the other thing. And just like, full transparency. To be honest. When we try to sell advertisements on this podcast and in our newsletter, the prospective advertisers are like, well, what can you tell us about our. About your audience? And right now we're like, well, literally nothing. We have no idea. So we're. We're asking people to sort of give us, like, general information about the types of jobs that you have or like the types of products that you like, things like that. So, yeah, it's voluntary survey. It would be very helpful. It's anonymous, of course, and trying to just like, get a sense of what types of people listen to this podcast and read our website and subscribe to our newsletter and that sort of thing. And then, yeah, on the ad front, it's like, if you are paying subscriber of 404 Media, you will not see ads. There's no ads, including on this podcast.
- [03:35]
Joseph
Yeah. So a link to that survey is in show notes. It'll take literally a minute, I think. And we would really, really appreciate it because it would help us grow sustainably and figure out, well, where exactly do we grow and where should we invest our time and resources? So let's get to the first story of the week. This one is written by Sam. It's a really, really crazy one. The headline is massive leak shows erotic chatbot users turned women's yearbook pictures into. Into AI porn. Sam, first of all, what is this service exactly? It is called Secret Desires. AI. What is it?
- [04:17]
Sam Cole
So it's really similar to a lot of the erotic roleplay chatbot slash image generator apps that are out there right now. There are a ton of these. You've probably seen them being advertised on Instagram or YouTube. They're pretty popular, all of them. They all have a ton of users. This one is really similar to, like, character AI Chub. AI is another one. It's similar to the meta chatbots, if you recall those from when I wrote about, like, the. The therapy chat bots that were on meta. Basically, you can go in there and say, I want to customize my own chatbot. Usually. I mean, it's called Secret Desires with like, a heart. So it's the main thing that they do is like, quote unquote, spicy AI chatting is what the website says. The site says, build your perfect AI partner. Customize their looks and personality to bring your fantasies to life. And it's like, it's, I would say, 99.9% erotic or, like, sexualized.
- [05:24]
Joseph
It's a Sex chat box stuff.
- [05:25]
Sam Cole
Yeah, I mean if you go through the, the homepage you can sort it by like 18 plus. It's, it's all like porn categories basically. It's like you can choose like by ethnicity, ages, realistic, anime, both. You can search by like kink. It's just like anime girls and fantasy women are probably 80% of the homepage, I would say. And then there's like ripped men. Like eight packs are also part of the offering, which I do want to, I mean I do want to say off the bat that women use these services. I would say not, maybe not just as much as men, but there's a, there's a big demographic of women using erotic chatbots and erotic role play because a lot of it's like text based and I think it's a lot of the same, not the same people but like the same kind of draw as like a romantasy novel except you're interacting with the character. So yeah, it's not like it's just men doing this, but it is on the homepage like mostly hypersexualized anime women.
- [06:33]
Joseph
And what do you see? And again, we'll get to the leak and stuff in a minute. But when you use Secret Desires AI, so it's a text interface, you're interacting with this AI Persona. But there are generated images as well. Is it like what, it's all part of the same product or what is.
- [06:51]
Sam Cole
Yeah, pretty much. So if you go through like the, so you can pick from the stuff on the homepage that I just described, um, there's like a bodybuilder lady who's like basically nude. There's like a sassy 50 year old pharmacist who is tender and nurturing and definitely not wearing anything work appropriate. Um, so you can pick from those or you can make your own, which I think is what a lot of people do. Um, so. So in the character creation part of it you can choose gender, you can choose style, like anime or realistic, you can do like identity. So you're doing like their, like their ethnicity basically, which there is not a lot of options to choose from there. You can do appearance, but it goes through all of this and it creates the personality of the chatbot based on what you choose. It's like you can choose their career, their traits, their lifestyle, you can type in your own prompts, but then you get to a point where you can then generate an image of your AI partner.
- [07:53]
Joseph
You make it basically.
- [07:55]
Sam Cole
Yeah, you're describing it, you're doing prompting just like you would with Sora or Nana Banana or any of these other stable diffusion. Any of those other generative AI platforms where people also use, you know, to. We've talked. Emmanuel's talked a ton about Civitai. So it's all part of the same universe and then.
- [08:16]
Joseph
Or you upload your. Your own image as well, right?
- [08:19]
Sam Cole
Well, not anymore. I mean that's. That's a feature that they used to have was. They called it face swap, but as far as I can tell, you were uploading an image of someone and it was gener. It was generative AI. It wasn't like deepfakes where it's a. It's a picture of someone and you're putting them in a scenario that already exists in real life. It's generating a whole new fantasy, a whole new person. Like who looks like the person that you are, the target person, basically.
- [08:50]
Joseph
Yeah, it's. I mean, I feel like Obviously readers of 404 and listeners of the podcast will know now, but like we are a million miles away from deepfakes at this point. Like, deepfakes are this quaint thing at this point where we're not talking about really face swapping, we're talking about using generative AI to make entire new scenarios and that sort of thing. Sort of like the Taylor Swift videos, which we might get to in a minute. Right.
- [09:17]
Sam Cole
And people still call all that like colloquially deepfakes because it's a person in a scenario that they were never in. I think is now the definition. The definition changes every year. I've given up trying to keep up on what, what specifically makes a deep fake. But I think most people are. When they say deep fake, what they mean is like all of the above. It's a huge umbrella term of like people generating wholesale new images, like these fantasies that people are making on Secret Desires and other platforms on Civitai, things like that. And then also the face swapping that you're talking about. So that was with like generative adversarial nature networks, which is a different sort of family of the same product or the same alp, I guess, without getting too like technical about it. That's what we're dealing with.
- [10:06]
Joseph
So that all exists secretdesires, AI offers that you then get this tip, I think via email. What exactly was this tip? What did it say?
- [10:18]
Sam Cole
So, I mean, there are websites out there that track open like exposed containers. And by container, I mean this is something that like, it's. It's storage, it's cloud storage being used by a company or by Even an individual. But usually it's companies because this stuff is expensive. And then they're drawing from that container to the, the inputs go in, they draw out of it. It's. It's cloud storage, AWS buckets, that sort of thing. Yeah, exactly. And so this specific, specific container was Microsoft Azure Blob Storage, which is such a funny name. Every time I come across it, that is what it is. It's a blob of data. And then the blobs are. There are different blobs and different containers of related data. So someone emailed me and was like, hey, I came across this on one of those sort of websites that tracks these things. He was like, this seems like a big deal because it's. Well, first of all, it was like 1.8 million images. A lot of those are duplicates, but it individual images, 1.8 million. And a lot of them were pictures of people, real people generated like AI characters and companions all coming from secret desires that was there. The container belonged to them. So this person flagged it to me and was like, this is something that you cover. This is probably not good, or at least interesting. I don't think I even think they said it.
- [11:46]
Jason Kebler
Not good.
- [11:47]
Sam Cole
I was like, yeah, that's super interesting. This was like on a Saturday or something. I was like, yeah, let me spend like the afternoon dragging these images out of this container to see what's actually in here.
- [11:58]
Joseph
Yeah. And just to clarify, again, this is publicly exposed information. Like it shouldn't be obviously, but it's basically been put in a database which is publicly accessible from normal Internet. So journalistically we are able to see the information and use it. And we've done tons of stories about exposed buckets over the years, not just here, but way back at Vice and Motherboard as well. So you go in and you get these images and it sounds like they're split into two sets, which is sort of the input where people are taking images of people and putting them in. And then the output like this AI generated non consensual intimate imagery or however you wish to describe it. The output first on the input images, what was in those? Is it like porn stars, celebrities, normal people? Like what were you seeing in there?
- [12:56]
Sam Cole
So yeah, and also just to your note that it's public, if you were responsibly storing this data, if you were responsibly using Microsoft's Blob storage, you would password protect it, you would encrypt it, you would be doing things to make sure people can't just access it from the open web, which is something Secret Desires obviously did not do, which was why I was able to just click on a link that took me straight to like a file full of this stuff. So what was in it? I mean, so, yeah, there were, there were. I guess that's. That's true about. There are two different sorts of images in there. There's the input and then the output. That's kind of the broad categories. But then there was also like specifically named containers. So one of the containers was named Face Swap and that's where a lot of the images that were kind of the. The crux of the story were being held. So a lot of those were just images of. From what I could tell, completely like random people. People would know like Internet and social media footprint. A lot of them were old images, like images from clearly from like 10 years ago, 20 years ago even, because they were like taken with flip flones, flip phones in mirrors, or with like a BlackBerry with the sliding keyboard. And they looked like deep fried, low resolution, like MySpace picture.
- [14:22]
Joseph
So somebody's gone and got MySpace pictures and fed them into this AI.
- [14:26]
Sam Cole
Yeah, and the name of the container is how we figured that out, because they're called faceswap. Faceswap is a tool that Secret Desires used to have where you could upload these pictures of someone, anyone, and then it would. I assume this is how it works just based on what we saw, based.
- [14:48]
Joseph
On how Face Swap works.
- [14:51]
Sam Cole
Yeah. People would then upload the images, it would go to the face swap container which was a public again exposed, and then that's where they were stored. And then Secret Desires was using its generative AI algorithm to turn those into like whatever fantasy prompting that the user described. So if they, if someone was like, I want to see my this. And this is something they advertise publicly as of like recently. If you want to see like this girl that I saw at the gym, um, that's a fantasy that you can fulfill by. It used to be by uploading a picture of this girl at the gym, maybe you know her like Instagram or like you took a picture of her without her knowing or whatever. Maybe she sent you a picture doing, you know, whatever you want. So like, I think we can infer. Infer based on the. The other containers which were full of porn, AI generated porn that people were making porn out of these people's images. So yeah, it was like regular people. It was also a ton of influencers, a ton of just like screenshots of influencers and also like sex workers and like porn influencers and adult content creators. Normal, regular, like non adult, non, not safe for work influencers, celebrities. There was one, probably the one that like really disturbed me the most was. And this is all, it was all pretty disturbing just to go through like these people's images who had no idea clearly that they were in this. But one of them was like a local, like a representative, like a, like a lower level politician who was. She was. The image was from where she was speaking at like some kind of like town council sort of event, city council thing and someone uploaded that and then you could go to the other container which said like what were the container names specifically anyway? They were, I think they were called like live photo or like character creation or something. But you could go to the next one and see where their faces looked a lot like the people who were in the AI porn. So I could, I found one of her looked just like her in like a really crazy like non, not humanly possible type porn. So yeah, it's like. And someone actually did end up telling me after this story came out that they were a tester for the face swap tool and they said that Secret Desires never let people upload those images from face swap to the public feed. But you could make them, you could put them wherever you want. Like just because you can't put them in the same ecosystem in the feed doesn't mean you can't just like share them in many other forums that these people also share things too. So it's a huge ecosystem and it doesn't just stop at like you can't upload it to the same website.
- [17:47]
Joseph
That point on public I think is like the really interesting bit about this piece which is that of course we see AI generated porn of people either on tube sites or it will happen with the Taylor Swift videos that go viral on X or whatever. But there are people doing this by themselves basically using these tools. And of course those are supposed to be opaque systems where I'm going to go to this website, I'm going to use this app, I'm going to upload imagery, I'm going to make AI porn of this person without them knowing. And maybe they don't distribute it, maybe they don't send it to the person, they just keep it to themselves on the hard drive. And of course nobody gets visibility into that. I don't think a lot of these people are going to want obviously visibility into that. You even though they're doing this to these other people. And this provides a snapshot of oh holy shit. This is what people are doing. And yes, there were a lot of responses going, are you really surprised? Like, no, of course not. But we're journalists, that's the point. So we found an actual use case of this happening and now there is a really significant data point to show. Just very, very briefly before we move on to the next bit. The headline does mention yearbook photos and I think Emmanuel came up with the headline. I think it was very, very smart to include yearbook because people instantly go, oh shit, that's a really horrible recontextualization of a photo or a piece of data. How did you figure out they were yearbook photos?
- [19:21]
Sam Cole
I mean they were literally like pictures you would see yearbook. It was someone graduating. They had class of 2010. I think it was like four or five or something. Like they were like holding their little cap and like it was like the life touch background. I don't know. Describe that with like blue speckley.
- [19:40]
Joseph
I think people can visualize.
- [19:42]
Sam Cole
Yeah, it's like you like, you know what a yearbook. And also I don't know if people know that like most. I think a lot of especially public school yearbooks are publicly available.
- [19:52]
Joseph
Right.
- [19:52]
Sam Cole
That's all data that's out there anyway. But this is like a specific person that obviously had access to someone's like yearbook picture. So maybe it was their classmate. It was just like some random person. But yeah, and then there were others that were like. There was a woman like standing in front, she was like graduating from university and she was like standing in front of like the sign for the school. Just like a very normal like wholesome picture. And someone took it and turned it into porn. Yeah, I don't know.
- [20:22]
Joseph
That's. That's not get too specific on the porn that was made, but I think can you give people a very general sense of how explicit or hardcore was this pornography?
- [20:38]
Sam Cole
Yeah, yeah. I mean and we've written about this before and it's something that is not new in AI, but with generative AI especially. So we're talking again about the difference between like old fashioned deep fakes where you're putting someone into a porn scene that already exists, which is already shitty and sucks because you're stealing someone else's content and also someone's face. But now it's like you can create scenarios and have people do things that just don't exist in real life. A lot of it was like pretty, I don't want to say violent, but like it was pretty violent because it was like not real people. It's like no one in this Is like the people are not real and also the person whose face it is is not consenting to like this being out there of them. There were quite a few very young generations like like generative AI of like very young people or what look.
- [21:46]
Joseph
Or what looks like a young person.
- [21:49]
Sam Cole
Like clearly the prompt and a few of these prompts were. You could see the prompts in some of the file names, but people were prompting for like 17 year old books like this and that celebrity. And it's like, that's a minor but like I'm not even talking about 17, I'm talking about like child.
- [22:07]
Joseph
It felt like a real mix because sometimes like with Emmanuel's Civitai coverage that has been specifically focused on child sexual abuse imagery, See Sam and then this, it sounds like had both potentially csam, but then also definitely adults. And it was like everything. And I was kind of taken aback by how explicit some of the images were.
- [22:31]
Sam Cole
Right. And AI generated child sexual abuse material in many states is treated the same. So generating porn like a sexual abuse imagery of a child with AI doesn't mean it's not a crime in many places. And experts have talked about how this is not only not helpful to like actual investigations and like tracking down people making this stuff, but also it actually damages investigations that are trying to find real like abuse victims out there. So I don't know, it's like that it opens up a whole other can of worms about like the, the conversation about AI and ccm. But yeah, a lot. I mean it's as far as like the, the hardcore porn stuff, it was just like like crazy crazy. I don't even know what's.
- [23:20]
Jason Kebler
Like the stuff that I saw when editing this piece was extreme. Like extreme almost all of it. And like, I don't like gang bang stuff. Like, I don't know really just like quite extreme. Which you know, is, is in the piece. But this is not like a standard nudify app where it's like taking the clothes off of a shirt off of someone who's wearing clothes. It is like for hardcore porn primarily at least what, what we saw.
- [23:53]
Joseph
Yeah. And Jason, I was going to bring you in now because you did. And I feel like when Sam got the tip, I think you were the first to respond saying something like, well, this is insane and wild. What did you think of this while you were editing it? Like, what's new about this story?
- [24:13]
Jason Kebler
I mean, we knew that this was the case, but it is another example of the fact that these tools are being used primarily to non consensually put women into porn and sexual situations, and it was not okay when it was deep fakes for either the porn performer whose body was being used or for the celebrities who were having their faces swapped onto was not okay then. But I think it was. It's, like, pretty alarming to see that quite clearly what's happening here is just, like, random people are being put into these situations where they're definitely not public figures and they're definitely not, you know, I don't know. It's just like, it's. It's pretty gross to see. And I think that we've done a lot of reporting where it's like, this is what these tools are being used for. But I think to see this scale at which this occurred and the explicitness and. And all that was, like, quite jarring. And then, yeah, again, it's like, this is not even a big one. Like, this is like, not. I had not heard of this app before. And the fact that there's more than a million images on this app is like, well, there's many, many apps like this. And I guess it's just like, I hate to be hyperbolic, but it's one of those moments that makes you, like, lose a little bit of faith in humanity, I think. It's just like, walking around, it's like, people are doing this. Like, people are doing this. That's not good. And we're not really doing anything to. To stop it.
- [25:50]
Joseph
And it could be you or your partner or a friend or something, because apparently people are just harvesting photos that. And they can. They're harvesting any photos they can get a hold of, essentially, is what it looks like. I guess just to wrap up this section. Sam, what was your takeaway? For me, it was sort of the scale and it was being done to ordinary people. Like, what was your takeaway after doing this?
- [26:17]
Sam Cole
I mean, not to like, keep belaboring the point, but, like, it's. It's just something that, like, we. We know people have been doing with the technology because, first of all, it's wildly popular. We had, you know, we know that erotic role play is hugely popular, and we know that generating real people is massively popular. And we know this from, like, lots of other reporting that we've done on this, and we know it from, like, listening to and, like, talking to the users of these tools. Like, they say it out loud all the time, that this is what they're doing with it. They're in discord talking about it and making these generations. It's like, that's that's. Whatever, that's fine. You know, like, fantasy is fantasy, porn is fantasy. You know, I don't like, I don't love that use of AI but go nuts. It probably shouldn't be illegal to make fantasy AI porn. But I think with the face swapping stuff, I think it was really, like you said, very telling that this is like a huge part of. That was a huge part of the site. It was also very telling that they took it down and that it was not a tool on the site for very long. It went down in, I think, April, and a lot of people were mad. A lot of people were like, I'm canceling my subscription. People were posting on Reddit talking about how they don't want to use secret desires anymore. And they were like, oh, it was my favorite tool, but now I've got to go somewhere else because they don't let me do face swap anymore. So, like, there are a lot of things going on there. It's like, first of all, people really want this feature. It's in demand. And also it was in demand and the company still took it down, which is interesting to me because usually these companies are only driven by, like, what did they find? Profit? What did they find? What did they decide, like, what. The Take It down act went into effect earlier this year. So, like, maybe they were like, oh, we have to like, preemptively make sure no one's violating federal law. So I don't know. It's like the company hasn't replied and said anything.
- [28:22]
Joseph
Which I shut down the buckets though, right? Or they closed them.
- [28:24]
Sam Cole
They shut down the buckets immediately, like in less than an hour. They, like, read my email and then shut down the buckets, which I sent them all the links because I was like, you need to shut this down because it's violating these people's privacy.
- [28:36]
Joseph
Or, you know, I was like, you can't cover it. Yeah. You can't cover it before it gets shot.
- [28:40]
Sam Cole
Yeah, we can't like, highlight something that's like an exposed. It's. This is something that's like, it's with any kind of like, security or like breach reporting. It's like you want the company to close the breach before you cover it because you don't want to draw more attention to it. So they shut it down. But, like, they didn't reply to me. And I would think it would be very easy to say, like, we found out that, like, it wasn't. We. We weren't into it. It's not what we're about. We don't like that use of it and we found out people were making like. But it is what they're about because that's all their ads are like AI women can't say no. You can generate the lady at your gym. That is what they're very much about. So I'm very curious. It's kind of like an open question about why they took it down. That's my, my takeaway is like a what's going on there? So I don't know. I mean, maybe we'll find out down the line. Maybe someone will come forward and say I worked in Secret Desires and I, I told them to take it down or something.
- [29:35]
Joseph
But yeah, yeah, if you're listening, you do know more, of course. Please reach out to Sam. We'll leave that one there. When we come back after the break, we're going to talk about one of Jason's pieces about some very funny that happened on X. But you know, there's much broader implications as well. We'll be right back after this.
- [30:05]
Jason Kebler
Are you always searching for that one hoodie you'll actually live in? The one that feels just right on the couch on a plane or out on a hike? I found mine with paka and now it's the only one I reach for. Pakka makes performance apparel from Alpaca fiber, one of the world's most sustainable natural fibers. Their best selling hoodie is softer than cashmere, warmer than wool, and breathable. This hoodie is built for real life. Thermoregulating, odor resistant, durable and made to last. It's lightweight, but still cozy, doesn't stretch out, doesn't pill, and somehow keeps me warm when it's cold and cool when it's hot. Basically, it adapts to wherever life takes you. Over 100,000 people have already picked up the paca hoodie. What makes it even better? Knowing it's made sustainably and ethically from traceable alpaca fiber while supporting the communities and artisans in Peru who bring it to life. Little known fact, it actually gets cold here in LA most nights. Actually, my paca hoodie has become my go to around the house at happy hour drinks and running errands. The only problem is I have to keep telling people where I got such a great hoodie from. If you've been thinking about leveling up your hoodie game, this year's sign to do it now. And right now, during PACA's Black Friday Cyber Monday sale, you can get 30% off with the purchase of three items or more head to go.pakaapparel.com 404media that's go paka apparel.com 404media to get 30% off with the purchase of three items or more. With Aura Frames you can relive your holiday traditions. And every day we've all got them. Those weird family traditions that make the holiday feel magic. For my sister and me, it really isn't Christmas until we put our childhood ornaments on the tree. We've got a lot of great pictures at that moment and Aura Frames is the perfect way to show them off. Aura Frames is a digital picture frame you can load with unlimited free photos and video. You can preload it with photos before it ships so you're ready to relive your holiday traditions the moment it's out of the box on Christmas morning. And the moment my sister and I put our ornaments on the tree this year, we can grab a picture and send it right to the frame from our phone. That way when our mom opens it up, she'll remember. Memory keeps these traditions alive and Aura Frames is the perfect place to store those memories. You can't wrap togetherness, but you can frame it for a limited time. Visit auraframes.com and get $45 off Aura's best selling Carver mat frames name number one by Wirecutter by using promo code 404 Media at checkout. That's a U R A frames.com promo code 404 Media this exclusive black Friday Cyber Monday deal is their best of the year, so order now before it ends. Support the show by mentioning us at checkout. Terms and conditions apply. I remember how shopping for glasses used to be stumble into a stuffy doctor's office, spend hundreds and hundreds of dollars on glasses I didn't really like, wait a month to get them, then feel dizzy because they weren't fitting right. Warby Parker changed all that. Warby Parker uses only premium materials in each frame. They designed the frames in house and their collection includes different silhouettes, colors and fits to suit every face and every fashion style. I love that it's a one stop shop. They offer everything you need for happier eyes, eyeglasses, sunglasses, contact lenses and eye exams. You can shop with them online and in stores. Warby Parker has over 300 retail locations across the US and Canada. Their glasses start at just $95 and include prescription lenses with anti reflective scratch resistant coatings. Add a pair and save 15% when you purchase two or more prescription pairs of glasses or sunglasses. This offer is available both online and in stores free shipping and 30 day returns. I've been going to Warby Parker for years now. I actually don't think about shopping anywhere else for glasses anymore. If you don't have time to get to the store, you'll love their virtual try on system that lets you figure out which frames and what size you need. Gone are the days where you'll be disappointed in your glasses. I've been lucky because it always feels like I've lived around the corner from one of Warby Parker's many convenient locations. Warby Parker has over 300 locations. To help you find your next pair of glasses, you can also head over to warbyparker.com 404 Media right now to try on any pair virtually. That's warbyparker.com 404 media warbyparker.com 404Media.
- [34:42]
Joseph
All right, and. And we are back. Jason, this is one you wrote. The headline is America's Polarization has become the World's side Hustle. Really good headline. Before we get into sort of the meat of your story, what was this change on X over the weekend? Because it, I mean it was pretty funny.
- [35:00]
Jason Kebler
Yeah. So X changed. They added a feature where basically you can click into someone's profile and look at what country they're based in. I'm not sure exactly how it's doing that, like whether it's IP address, whether it's like where it was registered, things like that. There's been actually some beef about this because I believe the Department of Homeland Security's Twitter page was showing that they were based in Israel, which I don't know if that's even real or not. It's like hard to tell what's real.
- [35:31]
Joseph
On X and I know but. And there's also like Hank Green's profile, the famous YouTuber. It says his was in Japan and like he's never been to Japan. Some of them look accurate. I presume it's IP and the registration one's easy enough. Like when they made the account, where were they located? That's straightforward.
- [35:48]
Jason Kebler
Right. So anyways, they added this feature and then people started looking at different really popular mega accounts. So there's one called Ivanka News that had over a million followers and that one was based in Nigeria. Uh, there's one called Red Pill Nurse that was from Eastern Europe, I believe, something called Maga Nadine which is in Morocco. An account called Native American Soul that was in Bangladesh. And there was many, many, many. It's like the entirety of my timeline over the weekend was about this and I don't know, I don't even go on X very much at all anymore. But this had like broken containment. I saw it, saw people in blue sky talking about it. I saw people on threads talking about it. And then so I went and looked and there's been like kind of a bit of meltdowns on both sides of the political spectrum here being like, well everything is just a psyop and every like, you know, all these accounts are disinformation campaigns, so on and so forth.
- [36:52]
Joseph
And what do they mean by that? As in just to hear their argument?
- [36:54]
Jason Kebler
Well, they're just saying that like a lot of these accounts are like, oh, like I'm a, I'm a single white mother from Oklahoma and I hate, hate woke, like the woke left and things like that. And that's like all they tweet about. And then you check, click into their profile and it's like oh, they're from Bangladesh or whatever. And so people are saying, well this is either foreign governments attempting to, you know, like divide Americans or it is like foreign interests, like big moneyed interests trying to do this. Or it's just like troll farms or bot farms, Dead Internet theory, like all this, that's sort of where the narrative went. And I guess before we get into it, let me just say that after 2016 election on Facebook, where there was these Russian bot armies and Cyprus fake news organizations and things like this, Facebook launched a feature that lets you see the country of origin for, for different pages. Now you have to click through like three different things to get to it. But if you do that, you will see like the vast majority of, I mean many, many of these like spam accounts that are posting like AI slop or like weird stuff say that they're based in Cambodia or Vietnam or just like other countries and they'll say like I'm a maga mom, like that that's sort of what they're, they're fronting as. And so this is not a new feature like for social media. This is something that has existed on social media for almost a decade, first of all. Second of all, I would argue it hasn't really done anything to better our online discourse or to make people understand that not everyone is telling the truth on the Internet about who they are.
- [38:47]
Joseph
You don't think the change on X may have some impact where even like, I don't know, very brain dead X users might be like, oh actually those big viral accounts I followed are from America. I mean may, I don't know, I.
- [39:04]
Jason Kebler
Think It's a good feature, and I think I'm glad that the feature exists. I'm just saying it's like 10 years too late and that YouTube has had a similar feature for a long time for channels where you can see where they're located. And it's like these things can be gamed, you know, with VPNs and with like, selling and buying accounts and things like that. But it is good and I think it's helpful. Like, it's helpful as a journalist so that we can go in and. And, like, look at where the country of origin is for some of these things, especially when some of them are claiming to be, like, official accounts or, you know, the, you know, verification on Twitter has been just a thing that you buy for since Elon Musk took over. But. But, like, it is a good feature. It hasn't done that much on Facebook to, like, fix the misinformation problem. And that's largely because this stuff is decontextualized when you look at it. Like you're just scrolling your timeline and someone has put it in your feed. And like, I don't know, there's. It's only a couple clicks, but. But how often are people doing the few clicks to sort of get to that on someone's profile? That said, it's a good feature, and I think it's led to interesting journalism. It's led to interesting. People have sort of woken up to the fact that a lot of these accounts are just trolling for engagement and trolling for clout and that sort of thing. But I don't know that is going to, like, heal our divide. And I guess I would also argue that it's like the divide goes further than a few weird mega accounts that are promoting things. But. But that said, it's like it's a new feature that people are talking about on X.
- [40:50]
Joseph
Yes.
- [40:50]
Jason Kebler
And it was, like, quite funny to see some of these accounts and, like, where they're supposedly based.
- [40:57]
Joseph
I mean, it caused a shitstorm, basically, where it looked like a lot of these accounts or people who followed them freaked out essentially, and just turned the entire weekend again. I don't really check X that much either. But then when I heard rumblings of this, I logged in and it's just like, wow, this is all anyone is talking about. And it's really, really good.
- [41:21]
Jason Kebler
Yeah. I will say before we get into my specific piece, that when Facebook did some of this stuff and when they've tried to make, like, try to do a little bit of transparency around this the sort of free speech warriors have said, like, this type of transparency is censorship and that sort of thing, which is really funny. But then when Elon Musk does it, it's like, oh, he is cracking down on bots and things like that. So to see the dichotomy of how these sorts of features are received has been interesting. But. Yeah, but you approach it from a different angle.
- [42:02]
Joseph
So that all happens over the weekend. We come back in on Monday. Right. And you then write this. And as I said, you approach it from a slightly different way because you've been covering very much the money and the ecosystem, specifically on social media behind AI slop. But not just AI slop, just almost like grifters in general now. It's just they're using a lot of AI right now. So what did you do to get this article going? It sounds like you started to search for certain things after you saw the X stuff.
- [42:37]
Jason Kebler
Yeah. So I did a series of posts about where Facebook AI slop comes from and I did. I found a lot of YouTube tutorial videos about how to make AI slop, how to bypass, you know, Facebook's pretty shitty filters and how to monetize your account and that sort of thing. And that was about a year ago and we talked about on the podcast. But something that I remember from that and something that I knew already was that there was a specific interest in targeting American users, like in getting in making stuff that goes viral in the United States. And the reason for that is twofold. One, we. Well, the reason for that is because advertisers pay higher ad rates to reach American users. They pay higher cpm, which is cost per millia or milli milli, which is cost per 1000 views. It's an advertising. It's how sort of like the advertising industry does rates. And they pay the most to reach American users for two reasons. One, because the United States has a relatively wealthy user base, at least definitely higher than like Pakistan, India, Cambodia, Vietnam. It's. It's like more valuable for people based there to make content that reaches American users versus people in their own countries. And the second reason, which is not in the article, but which Cory Doctorow texted me about afterwards and was like, you should have put this, and I should have to be clear, is that the U.S. yeah, the U.S. has like no privacy laws basically whatsoever. And so you can target Americans a lot better. You know, we don't have GDPR and things like that. So the ads are usually more effective because you're targeting people more directly and therefore the platforms can charge more money for them because they're, they're more effective. So for those two reasons, the United States is the most valuable ad market. And so a lot of the YouTube videos were like, you need to make content that is popular with Americans. And so when I saw this, I was like, I wonder if they're talking about this. I wonder how this like ecosystem has developed since I last wrote about it. And so I started searching for like monetized X account in Hindi. So I just use Google Translate. And I, I started in Hindi because it seems like India has the biggest ecosystem of side Hustler podcast bros. That's.
- [45:14]
Joseph
What you found before with your Facebook stuff.
- [45:16]
Jason Kebler
It's what I found before. And I mean there, this is happening all across the world. Like I found some in Cambodian, I found some or in Hindi and Portuguese and Vietnamese and then a lot in English, to be clear. Like people in the United States are doing this also, like using AI to spam social media. But it's a particularly popular in India right now. And I found so many videos from the last few months about channel ideas for what they call a USA channel. And it just means like content that Americans are looking at. And so these include ideas such as make content about MMA fighting, which this one Pakistani created creator said is similar to cricket in Pakistan in terms of Americans, like love this, they love mma. So make MMA AI videos. You know, make videos about American politics, make videos about cool cars, make videos about pets and dogs because Americans love dogs, things like that. And it's not like, yeah, exactly. It's not like there's like one or two videos like this. It's like there are dozens and dozens and dozens of videos about how to make make USA content, USA channel. And a lot of them are pretty basic, but then a lot of them are relatively sophisticated where they're like, they'll start off the video being like, CNN is a popular American news source. Go to CNN.com and copy paste their articles, put them into chatgpt change, you know, have it write you a script, run that script through an AI voice, generate images based on that script, and then, you know, use capcut and make a video and, and you can make money. You can like post this content and Americans will love it. There is also a lot of like black history content, which is kind of wild. And it's something that I've seen on YouTube a lot.
- [47:24]
Joseph
And it's popular or the people doing the grifting thinks it's popular, popular or both.
- [47:28]
Jason Kebler
I mean, people are trying everything it's like, it's like people are throwing at the wall. And so a lot of the tutorial videos are like go to Wikipedia, type in like Black history or African American experience, copy paste that, run it through ChatGPT or perplexity or like each, each, per, each like influencers. Like here is the type of AI tool that you should use and some of them have promo codes and things like that. So it's like it is a side hustle, grift, vibes going on. But that's like what they're promoting. And then they'll show you examples of the types of spam that has been created. And so like you know, they'll, they'll show like successful YouTube channels. And it's very interesting because we did a story a few months ago about like boring history videos on YouTube and there are a lot of these, a lot of these tutorials are like how to make boring history videos for YouTube. And so I mean there was, there was also some stuff about like how to, how to monetize an X account, how to monetize a Facebook account, how to monetize a TikTok account. So this is like the type of thing that is happening. Uh, and this is like why in my opinion we are seeing so much of this type of content that is like culturally off because it's not based in like I did deep research about what is occurring in the United States. It's very often like I use Google Trends and I copy pasted something or I like wrote a script in Hindi and then I translated it into English and then I used that to generate a script and there's like something lost in translation. And there's also something lost because a lot of these people are running a lot of different channels and so they're not or a lot of different social media accounts across different platforms. And so they're not necessarily like doing a deep dive into how can I be the most accurate. And so I think like my theory of the case is that yes, there are some disinformation campaigns out there. There are some like true believers who live in other countries who are, you know, trying to either sow division or who are like actually interested in the maga movement. There are obviously tons and tons of Americans who hold really polarized views in either direction who are making a lot of content about it and are beefing online about it. But then you also have the entire rest of the world who have been incentivized by these social media companies to make content that Americans will see and what are Americans talking about on the Internet right now? Well, they're talking about politics and they're talking about the economy and they're talking about tech news, and they're talking about ICE and immigration raids and all this sort of thing. So if Americans are talking about it in any way, there's going to be spam content about it. And these social media companies, like, they incentivize it both through their algorithm, which incentivizes division and engagement, and, you know, incentivize, frankly, like spam. A lot of you, you post enough like, you're gonna go viral. And then it also has these programs where it directly pays the people who post them. And I should have said that up top, but like.
- [50:57]
Joseph
Well, I was gonna ask it now, but you just asked.
- [50:59]
Jason Kebler
Yeah, it's like the, you know, X has a monetization platform where, like, if you enroll in it and you go viral, you get direct payments from X. And YouTube has, you know, an AdSense thing where you get a fraction of ad revenue. Facebook has a creator's program where, you know, they pay out and it's not. That's not directly ad revenue. Or, like, they wouldn't say that it is, but that's where it's coming from. But it's like they pay out based on views. We actually don't know the full kind of, like, formula for how they calculate. But a big part of this YouTube influencer economy is for them to show their dashboards where they show how much money they make. And they show very clearly, like, this video or this photo that I posted had 60% of its views come from the United States. And I made a lot more money than this other one where most of the viewers were in Eastern Europe or whatever. And so, you know, they've learned this and they're like, well, we're going to make content that theoretically Americans will care about.
- [52:06]
Joseph
Yeah. And you, when you have the social media platforms introduce these monetization strategies and programs, obviously that creates all of these perverse incentives where people are not posting for the sake of accuracy. They're not even posting for the love of the game, man. They're posting to get their money, basically.
- [52:26]
Jason Kebler
They're posting get their money. And I mean, here's. Here's one of the slides that I saw. And the slide is in English, which is interesting. Some of these videos are actually in English. A lot of them are. Are not. But now there's a lot of really good translation tools. But anyways, it says YouTube earnings Pakistan versus USA YouTube earnings in Pakistan $1 per 1,000 views. And then YouTube earnings in the USA, $5 to $7 per 1,000 views. And so, you know, you're making between five and seven times as much according to this video, if you're able to reach an American audience. And so why, why would you cater something to a smaller audience or like an audience that these companies are not paying as much to reach when it takes the same amount of effort to spam something that is about the NBA or Major League Baseball or something that. Or, you know, American politics, Something that is specifically focused on American like American audience and viewership and politics and news.
- [53:34]
Joseph
Yeah. All right, we'll leave that there if you're listening to the free version of the podcast and I'll play us out. But if you are a paying 404 media subscriber, we're going to talk about how we contributed to the shutdown of a warrantless surveillance program. You can gain access to that content at 404 Media co. As a reminder, 404 Media is journalist founded and supported by subscribers. If you do wish to subscribe to 404 Media and directly support our work, please go to 404 Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really does help us out. I'll read out more of those soon. This has been 404 Media. We'll see you again next week.