Loading summary
Transcript73 lines
- [00:00]
Sam Cole
Foreign.
- [00:06]
Joseph
Hello and welcome to the 404 Media podcast where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist founded company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to our with their best comments. Gain access to that content@ 404 Media co. I'm your host, Joseph and with me are two of the other 404 Media co founders. The first being Sam Cole.
- [00:41]
Sam Cole
Hello.
- [00:42]
Joseph
The other being Emmanuel Mayberg.
- [00:45]
Emmanuel Mayberg
Hello.
- [00:46]
Joseph
All right. If you're a new listener to the podcast, you would have just heard some fun, interesting things, funky cybernetic music. I'm running out of adjectives at this point. If you're a regular listener, you will of course also have heard the same music, but you would be thinking, whoa, wait, that's not the ordinary introduction to 404 Media. That's because we've replaced the very cheap stock music I bought very quickly when we were launching the company in 2023 with new music written by our audio producer Alyssa Midcalf. So really, really appreciate it. Love it. You're gonna hear more music for the ads, I think, and definitely the Outro as well. The outro is my favorite. Sam and Emmanuel, what do you think?
- [01:41]
Emmanuel Mayberg
We're pros now. We're legit. It really elevates the podcast. It makes us sound really, really professional.
- [01:48]
Sam Cole
Yeah, we had a bunch of comparisons to what we thought it sounded like and several of them were like, we need an 8 bit version of this. We had some good DOS games in there, I think. I was talking about DOS with Harlow, which is. She's on an interview episode with us in a couple weeks that'll come out. And we were talking about Ms. DOS and Linux and stuff like that. And I think it was on my brain thinking about that and the way that this new entry music sounds like. It's very like. What did you say, Emmanuel? It was like, it sounds like we're in an underground waterworld dungeon or something.
- [02:22]
Emmanuel Mayberg
It's like, yeah, we're getting into the dungeon level. Cave level. That's the vibe.
- [02:27]
Sam Cole
Cave level. Yeah, it's good.
- [02:29]
Joseph
Yeah, I love it. Another piece of housekeeping. Sam, There is a new podcast which came out today. We're recording this Tuesday. It comes out the day after for non paying subscribers on Wednesday. What is this podcast?
- [02:46]
Sam Cole
Yeah, so I've been working with the CBC on a podcast about deep fakes. The podcast is called Deepfake Porn Empire and it's in their Understood series. If you just go to your podcast app and search Understood cbc, it'll probably pop up. But yeah, we talk about the history of deepfakes, which goes actually way deeper than I even realized. I learned a lot just making this podcast with them. It goes back to my favorite places on the Internet, which are like Usenet forums. Back in the 90s, people were making, you know, like Photoshop and collage level quote unquote deep fakes. But it was like, you know, porn face swap stuff.
- [03:31]
Emmanuel Mayberg
Yeah.
- [03:32]
Sam Cole
And we also talk about the investigation that the CBC did into Mr. Deepfakes, which is one of the biggest deepfake porn sites out there, or used to be because it's gone now due to their investigation. But I got to talk to the journalist who. The several journalists who worked on that who confronted the guy who found the guy. He like lived in a suburb of Toronto and they like found him backing out of his driveway in his Tesla through like a overnight stakeout situation. It was, yeah, it was really, it was a really crazy investigation on their part. So I got to talk to some of those investigators from Bellingcat, from cbc, from Checked at a couple others. But yeah, yeah, it's a good time. Even though it's a really dark topic. I think just bringing in some of the targets of deepfakes especially and hearing from them has been really important and is a huge part of this podcast as well. So. And also the rest of the show, the rest of the understood feed is probably really interesting for four or four listeners. They had Cory Doctorow on for a season to host about insertification. They had Jacob Silverman on about Elon Musk. They had like a Celine series, which is so funny to me, kind of left field. But like, I haven't listened to it yet, but I'm absolutely going to, I guess because she's from Quebec. But yeah, it's a good, good series in general. But yeah, the one that I hosted is first episode's out today, trailer's out next episode two will be out next week.
- [05:09]
Joseph
Sure. We'll try to put a link in the show notes if it's out, should be able to grab that and point people to it. But yeah, definitely check it out every week, add it to your feed if that sounds like something you'd be interested in. As for this podcast, as ever, we have a bunch of stuff to get through. The first one is one that Emmanuel has been working on for a while. It's complicated, but absolutely Crazy, frankly. I remember when Jason read it today and pushed it out on Bluesky. He said, like, every bit of this story is insane and it does get crazier as it goes along. The headline is a quote, students are being treated like guinea pigs. End quote. Inside an AI powered private school. So let's set just a little bit of context. And Emmanuel, this is about something called Alpha School. Some people may have heard of that, I imagine a lot of people won't have, but it is a big deal and Alpha School has got a lot of attention recently. And of course that's one of the major reasons why we're covering it. Beyond obviously you doing all the reporting as well, to lay the groundwork. What is Alpha School and why is it a big deal?
- [06:28]
Emmanuel Mayberg
Alpha School is, as you said, an AI powered private school. What that means in practice is there is like an Alpha School curriculum or program. They have physical locations across the country. They also have some sort of homeschooling product that people can use if that's how they educate their kids. And kids go to these locations. They study for two hours a day, working on kind of like the core curriculum. This is the stuff that the government said that kids have to know, the stuff that will eventually lead them to the SATs or the Advanced Placement AP classes that kids eventually take in high school, the standardized tests, all of that is condensed into two hours. And then once that is done, the kids can focus on more enriching activities, stuff that they work on with their teachers and other students, whether it's they're starting a business or they're working on public speaking or whatever, just like enriching more social media, more involved, innovative ways of learning. The idea of condensing that core curriculum into two hours has existed for a while, but the Alpha School method is essentially to help facilitate that with AI, with generative AI. So broadly speaking, that means that they are AI generating classes or courses that kids can take, depending on their current level. And part of the pitch is that it's personalized, that is like a big concern for parents, is kind of the one size fits all system in public schools. And this is an alternative to that that is allegedly made better with AI. And AI has been hitting education really hard in all forms, college level, public schools, private schools. Jason wrote a really great piece, I think it was last year that was based on interviews with teachers, kind of talking about how AI is already affecting them. Alpha School has emerged as the leading example of how this goes well, how AI is incorporated in a smart, useful way. It's promoted by a lot of people in tech. It has Bill Ackman as a backer. Linda McMahon, who is the Trump appointed Secretary of Education, visited their campus in Texas and talked about how wonderful it is and how it's the future of education. I think just finally how to set this up is I think a really good way to think about Alpha School as a company is by talking about two of the people who are leading it. One, one is McKinsey Price, who is the person who is promoting this two hour learning philosophy. And she's been doing this since 2014. And then there is this other person who is the quote, unquote principal of the school. His name is Joe Limont. He is the person who's at the head of a quietly gigantic software company called Trilogy that does a lot of the development work for Alpha School, but also. Has created something called. I'll have to look up the name. It's a crossover, I think, and that is a remote work platform where companies can connect with remote workers. And the key to that company is what we call bossware. Just like very closely surveilling the workers to make sure that they're doing their work on time, that they're being efficient. And that philosophy has kind of made its way into Alpha School as well.
- [11:06]
Joseph
Yeah, so it's like that combination of this idea of two hour learning where you condense the learning into there and then you combine that with that bossware spyware and we'll get into more specifics in that in a minute. But combining all of that with AI, with the promise that you send your kid to this school, which costs more than $60,000 a year potentially.
- [11:32]
Emmanuel Mayberg
Prices vary, but it's most often 40 and can go up as high as 60,000 a year in California.
- [11:39]
Joseph
Yeah, so it costs a lot of money. That's the promise. It has received media coverage. It's not like we're the first outlet to cover this, but it's been of a different vibe. Right. They've been on the Hard Fork podcast by the New York Times just before we then move into our investigation. How would you characterize the coverage and the perception Alphascor had up until this point? You said it's popular in Silicon Valley. Is it popular in the tech press? Like, how do people perceive it? Do they think, oh, this is actually an interesting idea and like this could be fruitful? Like, what's the vibe really?
- [12:21]
Emmanuel Mayberg
The vibe is that it's swept up in AI hype and that is the.
- [12:27]
Ad/Promo Voice
Main.
- [12:31]
Emmanuel Mayberg
Sentiment of the coverage around it. People Talking to mostly Mackenzie Price and discussing how wonderful AI is and how it's helping students without many specifics, local news covering it, Fox News covering it, kind of all the tech influencing people on Twitter tweeting about it. I would also mention that our friends at Wired did a very good investigation about alphasca School, but that was more about how the close monitoring and kind of productivity mindset of the school was really burning out some students and leading to parents to pull them out. But the focus of my investigation is the AI product, some privacy concern and that is based on a lot of internal documents that show how Alpha School itself thinks about these issues.
- [13:36]
Joseph
Yeah, so we'll talk about the specific findings in a second. And it's a really detailed investigation, so I do encourage people to actually go read it themselves as well. But I've pulled out sort of three main areas and we'll get into those in a second. But you mentioned this material you received. What were these leaked documents and who did you speak to to inform this investigation?
- [14:05]
Emmanuel Mayberg
I talked to three former employees, but I think most critically is I got a snapshot of something called workflowy. Workflowy is another organizational productivity tool. It's basically a note taking app that you can share with as many people as you want. And at Alpha School, the philosophy was that everybody documents their work and shares it with everyone else. So it's kind of like a really, really, really long document that shows what everyone was working on at that time, what they were thinking about it, the problems that they had. They use this method where they call it their second brain. So it's like everyone is essentially saying in public, like, here's what I'm thinking about. The thing that I am currently dealing with, here are the problems, here are the proposed solutions, here's my strategy.
- [15:15]
Joseph
And.
- [15:17]
Emmanuel Mayberg
That also references a lot of documents where people are sharing feedback from students and examples of things going wrong. And I had some access to that stuff as well.
- [15:36]
Joseph
Yeah, so it's like a series of documents, but almost like a really helpful overarching document as well in this work. Flowy, I think is the name. And, and yeah, it gives you really piercing insight into how people inside AlphaSchool are actually thinking about the products and the AI they're making and deploying and the impact of that as well. So again, there's a lot in there, but here are sort of the three things I pulled out. Rather the first one, the faulty AI generated lesson plans. Obviously that is going to be probably the biggest concern to parents or to students who are actually At Alpha School. What is the deal with these faulty AI generated lesson plans? What's happening there?
- [16:28]
Emmanuel Mayberg
Yeah, so before I get into the specifics, I would just say that it's hard to overstate how bought in Alpha School as a company is on AI. They use AI for everything. This thing that I just Talked about, the second brain, everybody feeds that into ChatGPT and then talks to ChatGPT about their projects to get ideas and find solutions. But it also is how they generate a lot of the courses. They will digest a bunch of material, feed it to an LLM, and then essentially say, give me a class about this subject and generate multiple choice questions about this subject. The best example of this, I think is something that the company calls Alpha Read. This is an app that students access and work on their reading comprehension skills all the way from grade school up until SAT level reading comprehension questions. And I think the most damning example that is in the story is there's a multiple choice question about an article and the AI generated this question and the answer is illogical. Like none of the four choices that are given to students make sense. It's like they're asking them to complete a sentence to show that they understand what they just read. But none of the answers make logical sense given the context of the art or grammatical sense. Like literally you can't make it work. And in the workflow you can see employees talking about the fact that, that this is something that happens. It doesn't happen with every question. There are indications that these type of problems happen at least 10% of the time. Like hallucinations happen 10% of the time with all LLMs. And that appears to be at least as common here. And they talk about how this does quote more harm than good because you're trying to teach someone and you're telling them that this is the thing that they should learn, but the thing that they are learning is incorrect. And that creates a problem where students a get frustrated and then don't trust the teaching. And I don't know, just think about it as like, I don't know how you all were as students, but it's like school was pretty difficult for me as is. And the idea of me being presented with a problem that I literally can't solve because it's like it's been faulty generated by AI just sounds really, really frustrating.
- [19:28]
Joseph
Is the teacher gaslighting me or playing a trick or something? And as a child, I mean, depending on your age, obviously you may not know that. Oh no, it's because it was AI generated and it hallucinated. And why would you assume that? And also you would probably just assume that, well, this is a member, somebody in a position of authority, my tutor or my teacher or the school or whatever. Surely this question is going to be correct and you try to read it. I think we actually to the screenshot in the article and yeah, you can't pass it at all. It just doesn't make sense. I know you said the 10% figure and some people may hear that and be like, well, it's only wrong 10% of the time. That sounds like quite a lot for a school. And enough so that in these internal documents that you obtained, employees do really think is an issue. Clearly.
- [20:24]
Emmanuel Mayberg
Yeah. I mean the 10% of the time, which is a minimum. Right. That is just about the highest or the worst level of error where it's generating a patently impossible question. We have no idea how common it is, but the workflowy shows that it is a problem where the AI generates things that the students can answer. But they're not actually preparing them for the sats. Right. So they're taking a bunch of questions and reading a bunch of articles thinking that they're preparing for the SATs. But the comments on those materials from Alpha School employees show that they're actually not teaching them the reading comprehension skills, skills that they need. They don't require critical thinking, they don't require a deep understanding of the articles that they're reading. Like an example that I saw is that you can answer one of those questions just contextually giving the paragraph that they're asking you about. It doesn't actually require you to have understood the essence of the article that you just read, which is what you're supposed to learn in order to allegedly, I don't know, succeed in life and do well on your sat.
- [21:45]
Joseph
Yeah. And it seems like some students are getting frustrated and they then have to sort of go back and do further education because they haven't fully understood what they're actually supposed to do, all of that sort of thing. The last thing on these lessons before we move on to the other couple is that AlphaSchool also uses AI to review and critique those AI generated questions. So it's using the AI to fact check and sort of probe and try to improve the output of the AI, which we've already said can be faulty. So there's like some sort of cycle going on there, right?
- [22:25]
Emmanuel Mayberg
Yeah. So the company again, very bought in on AI and the future that they're striving towards is, is to Generate the entire lesson plan with, again, quote, no human in the loop. A completely autonomous AI system that is educating. That's what they want. That's where they want to get to. And in order to get there, what I heard from employees is that they're removing as many humans from the process as they can. And that means that students are presented with questions that are not vetted by humans and they're relying on AI to vet the quality of materials. And this is sort of like a classic problem with AI that everyone understands is not the correct solution. It's like you have an AI, it's generating some sort of material. You know that 10% of the time there are going to be hallucinations in it. And people before have tried to solve this problem with just saying, like, oh, let's take another AI or take the same AI and have it do another pass on a generated materials in order to fix the problem. Obviously that's not going to work because.
- [23:37]
Joseph
I just don't understand the thinking. I really, really don't. Like, maybe I'm missing something. I just don't get it. Why would you use AI to check the output of AI? I just fundamentally don't understand.
- [23:48]
Emmanuel Mayberg
Yeah, I mean, anyone who thinks critically about AI knows that this doesn't work. And there's been many examples in the past of people trying this and failing. But, I mean, yeah, that's what they're doing. That's what they want to do. They think they can get there.
- [24:02]
Joseph
Yeah. Yeah. Okay. Then you found that AlphaSchool is scraping the content of other online sort of education platforms and resources to then incorporate into their own lessons. But it looks like, based on the material, they're doing this without permission now. It's almost like a story as old as time at this point, where Sam did a bunch of reporting about the Nvidia stuff as well. We've done lots of reporting about building AI tools based on scraped or repurposed data, that sort of thing. So is AlphaSchool basically doing the same thing? What are they doing there?
- [24:45]
Emmanuel Mayberg
Yeah, very much. This part of the story very much reminded me of Sam's reporting. You see, again, people in internal documentation instructing other employees to use their personal email accounts to sign up for various online learning sources and scrape them. And the reason that they want people to use their personal emails and not the company email is so that they don't get banned. Right. It's like if, let's say, Khan Academy, which is a online learning platform.
- [25:19]
Joseph
Pretty big, pretty popular. Yeah.
- [25:21]
Emmanuel Mayberg
And which it seems Alpha School scraped extensively. If Khan Academy notices that they're being scraped a bunch by the same company email domain, then they could in theory ban them. I've seen people being told to buy materials if they need to and then expense it. I've seen internal documents that just show a ranking of like, what are the most valuable textbooks that we can scrape and those rankings showing, well, here's what we can get from this textbook and here's why we should scrape it, and here's how we could scrape it, including instructions on how to not get rate, limited on scraping a particular service and stuff like this. There's also again, Wired article last year mentions that ixl, one of these learning platforms, terminated its relationship with Alpha School for violating its terms of service. It wasn't clear why. The article doesn't say. So my piece found that Albert IO, which is another company that offers a similar service, terminated them also for violating their terms of service. So there's a pattern of them kind of going to an online platform, scraping it as much as they can, the company finding out about it and shutting them down and sending them a cease and desist or something like this. And they're doing it to online materials. They're also just doing it to text material, books, just whatever they can get their hand on.
- [27:08]
Joseph
Yeah, gotcha. Yeah. So they're very much training their own systems rather than just sort of shoving it into ChatGPT, even though they seemingly do some of that as well. And then the last part is sort of the surveillance of the children you mentioned the Wired did an investigation, sort of touching on that a fair bit. And then this looks at it as well. What did you find about the surveillance of these children? Like what is being monitored and how is that data being stored?
- [27:41]
Emmanuel Mayberg
Everything is being monitored. Again, Joe Lyman, the principal of the school, has this company called Crossover. It tracks mouse movements, it tracks what websites you're going to, it tracks what apps you're using, how much time you're spending on each task. All of that is also true at Alpha School. They do this, they say, in order to improve their classes, but they also do it in order to, I want to use a word that's not so harsh, but compliance. It's like to enforce their learning, to make sure that students are actually engaging with the materials. And one of the ways they do that is they record videos of the students so they can see is the student looking at something else, are they looking at their phone, are they getting up from their desk to do an entirely different thing. And one of the first things I saw when I started reporting this story, which I thought was really shocking, is it was just a spreadsheet on Google Docs that had the name of each student, what grade they were in, what classes they were taking, and each of those names linked out to a video recording of a tutoring session they did on Zoom. And that was also stored on a completely open Google Drive. Like right. When you share files on Google Drive, you can have them completely locked down, you can share them with specific emails, you can share them only with people within your organization, or you can share them with anyone who has a link. Right. It's like that's one of the most permissive permission is anyone who has a link can view this and that's how these files were shared, which is just like really irresponsible and possibly dangerous.
- [29:55]
Joseph
Yeah, maybe there are some legal ramifications there, which I didn't really think about while reading, but now just speaking out loud, maybe there's something there, but I obviously have to look into it. So you do all of that and the piece is very nuanced, it's very fair, it's very balanced, but it does highlight all of that really important stuff you just said. What was your main takeaway from reading all of this material and speaking to these employees? Is it something like Alpha School's approach is completely fundamentally flawed, or is it more like the marketing doesn't match the reality? What's your main takeaway after digesting all of that material?
- [30:38]
Emmanuel Mayberg
I think I've thought about this a lot and I think the findings in the article I think are pretty damning. I think presenting students with faulty AI generated questions is really bad. I'm just thinking of myself as a student in high school having to go through that. That sounds awful. The constant monitoring sounds extremely stressful. The scraping obviously is not fair to all these other organizations. So all of that is really bad. At the same time, I think sometimes we'll publish an article and I as a reporter at the end of that process, and I think readers at the end of the article kind of walk away and say this is completely irredeemable. Right. Like this whole thing from the ground up is really messed up and should not exist. Right. I'm thinking about a lot of websites dedicated to non consensual content or something like that. Right. This is not that I think, and I have a two year old and the state of education in this country is such that even when he's this young, we're already thinking about high school and what school Is he going to go to, Is it going to be public? Is it going to be private? Is it going to be some charter school, a magnet school school? It's like parents are very freaked out about education for their kids. All the research shows that it's like such a huge determining factor for the rest of your life. So I understand why parents would gravitate towards this school, which a does produce good results. Like, no one I talk to disputes the claim that Alpha School students are in the top 2% nationally in terms of their test scores. And if you look at that as a parent, I mean, that's very appealing. If you can afford that, sure, that's great. I think the other core philosophy, like this two hour learning philosophy, that also sounds extremely appealing. Like if you can condense all the boxes, you have to check for your education, for the sat, for the standard.
- [33:01]
Kylie Brewer
Right.
- [33:01]
Emmanuel Mayberg
Test for all this stuff to two hours and then free up the rest of the day for students to pursue their passions and find themselves and motivate themselves. Such a huge part of education is motivation and just finding something that you actually want to do. Again, very appealing idea. I say this with the caveat of trends come and go in education and people get really bought in in an idea and then we find out that it's bullshit. And also with the caveat that it's like I'm not there yet. Right? It's like, I don't know what it's like to have a kid at that age and deal with these questions, but these are all valid concerns and this is like a valid thing to pursue. And the employees that I talk to think this as well and are very passionate about education. That's why they wanted to talk about this with me. It's just that at some point, like many other companies, they pivoted to this idea that AI is a magic bullet for all of these problems. And that, at least so far, is patently not true. And they know it because they're talking about it internally at the company, knowing that it's not working yet, and they think they can get there. But actively, students are being presented with faulty AI generated questions.
- [34:31]
Joseph
Yeah, I think that's a very, very good way to pull it. All right, we'll leave that there. Please go read the full article linked in the show notes. When we come back after the break, we're going to talk about one of Sam's stories about. Well, I don't really know how to summarize except the headline. So we're just going to wa until we come back after this.
- [35:00]
Ad/Promo Voice
Most people think their heart health is fine because their cholesterol looks normal, but have you ever heard of something like lipoprotein A? I hadn't, and they often aren't tested unless you specifically ask for them. It's National Heart Month and I've been thinking more about heart health and long term prevention. I'm especially worried about staying ahead of problems before they start, and that's surprisingly hard to do unless you actually start looking deeper. That's what I like about function. It doesn't tell you what to worry about. It shows you exactly what's happening inside your body. Through a series of lab tests accessed through Function, you get a more comprehensive view of markers tied to heart health like inflammation, stress hormones and toxins. Once you can see the data, you stop guessing and start making decisions based on real insights. I finally took my function test a few weeks ago. It was easy to schedule and I was in and out of the lab in a few minutes and I got my results within a few days. I was shocked by how much more comprehensive it was than standard blood panels and it gave me the information I needed to help guide my own health choices. Function was designed by world class physicians and is trusted by hundreds of thousands of members. When you understand your health, everything changes. For me, it's about clarity. Seeing my results on the Function platform helped me focus on the habits that actually matter sleep recovery, stress management, how I eat instead of just assuming everything was fine. Own your health and start with your heart. Function gives you access to more than 160 lab tests and each year, including advanced markers for heart health, inflammation, stress, hormones and toxins for $365 a year. That's a dollar a day. Learn more and join using my link, visit functionhealth.com 404media and use gift code 404Media25 for a $25 credit toward your membership.
- [36:55]
BetterHelp Sponsor Voice
This episode is sponsored by BetterHelp. Look, I get it. February means Valentine's Day means flowers means thinking about your love life means anxiety means Do I have it together? Take a breath. It's okay. Everyone is still figuring things out and maybe you just need someone to talk to. That's what therapy is for. Whether it's for individuals or couples, therapy is an opportunity to identify what's getting in the way and help remove those blockers and help ease that anxiety. And BetterHelp can get you linked up with the right therapist for you. BetterHelp does the initial matching work for you so you can focus on your therapy goals A short questionnaire helps identify your needs and preferences and their 12 years of experience and industry leading match fulfillment rate means that they typically get it right the first time. And if you aren't happy with your match, you can switch to a different therapist at any time. Right now, four or four listeners can get 10% off. So so sign up and get 10% off@betterhelp.com 404 Media that's B E T T E R h e l p.com.
- [38:09]
Ad/Promo Voice
404 Media@ 404 Media we know your personal information is everywhere. Online data brokers crawl the web to build profiles selling your home address and phone number to anyone with a credit card. Trust us, we know this better than anyone else. This exposure leads to risks like doxxing and identity theft. That's why we recommend Deleteme, a hands free service that removes your personal info from hundreds of data broker sites. Their experts find and remove your information providing a detailed Privacy report within 7 days. DeleteMe monitors these sites all year long to ensure your data stays removed. We feel like we're at increased risk because of the types of stories we cover and the public nature of our job. But the truth is identity theft, threats and harassment can happen to anyone. I've been doxxed. In fact, lots of us have been doxxed. And each time Deleteme has helped us pick up the pieces, get our data removed from shady websites and otherwise clean up our online presence. I've been using Deleteme for years and every quarter I look forward to seeing what new interesting and horrifying websites they've had my data taken down from. It's kind of a perverse little game. Take control of your data and keep your private life private by signing up for Deleteme now at a special discount for our listeners. Get 20% off your delete me plan when you go to JoinDeleteMe.com 404Media and use promo code 404Media at checkout. The only way to get 20% off is to go to JoinDeleteMe.com404Media and enter code 404Media at checkout. That's JoinDeleteMe.com 404Media code 4:400 404Media.
- [39:56]
Joseph
All right. And we are back. Another quote headline. Actually the headline of this one is the most dejected I've ever felt. Harassers made nude AI images of her, then started an onlyfans. There is a ton going on in the headline, Sam. So take us to the beginning Sort of chronologically in January, Kylie Brewer, she is a content creator. She started to receive a bunch of very strange, ominous messages. What did they say?
- [40:37]
Sam Cole
Yeah, so she started getting DMs on TikTok and I think on Instagram and just like various places where people were messaging her and saying, asking if she has an OnlyFans account, asking if she had set one up and didn't realize it, or like, are there two accounts and one is pretending to be her, you know, saying, I think someone has made an only fans in your name. Here's a link. Some of them were men messaging her, asking to get access to an OnlyFans account in her name and asking, you know, why they couldn't subscribe or like, if they could, if she could send them content or whatever. And then a close friend of hers messaged her and said, I hate to tell you this, but there's pictures of you going around and they don't look real. They might be deep fakes or AI, you know, either way, I'm sorry. So she started getting these messages, obviously very weird, very concerning, and she wasn't really sure what was going on at first, but obviously someone had been making AI images of her.
- [41:48]
Joseph
Yeah, yeah, very confusing messages all around. She then figures out sort of what has actually happened. Can you lay out for the listeners now what had actually happened? Once she figured that out, what were these people doing? And sort of what was the connection to OnlyFans?
- [42:09]
Sam Cole
So this all started in end of December, early January. And if you remember what was going on at the time, GROK was in the middle of making all these images of women in clear bikinis, you know, doing all these, like, sexual poses that people were asking it to generate of women without their consent on X and from elsewhere, just posting women's images and saying, hey, can you take her clothes off on this picture? And then GROK would return an image. There was something like 3 million of these sorts of images going around at that point. There were, I think, let's see, 23,000 that appeared to depict children, according to the center for Countering Digital Hate. So it was a huge problem at the time. It was just like kind of in the thick of it with the GROK issue with the nunifying images. So it wasn't that hard to kind of trace back, like, okay, that's what's going on here. People are making images of her using GROK and posting them online and then also taking it a step further and creating OnlyFans with, with that content. So using those images, putting them behind a paywall and then trying to like scam people off of it and also harass her in the process. So yeah, I mean, it's just she was, she's a very like public person. She's a content creator herself. So she's used to harassment, especially like sexualized and gendered harassment online for many years. Definitely draws the vitriol of the manosphere, which would encompass like incels, red wing or red wing, red pill, right wing influencers and just people who are in that sort of cesspool. So this wasn't like shocking to her that she was being targeted, but it was definitely highly triggering and also just like the OnlyFans thing was just another step in a direction that she had never seen before in her time online.
- [44:21]
Joseph
Yeah, I think it's really like, it's really rare to see this whole. Sorry to pull it so coldly, but this supply chain basically, or like this sort of pipeline where. Yes, we all know that, especially because it's all been done publicly, as you say on Grok, that people are using AI nudify apps. We've reported a ton about how they're being used in schools and elsewhere. We've done that a ton. And we also know definitely through Jason's reporting that some people try to scam by running OnlyFans or other sort of like almost fake influencer accounts as well, where, you know, this person doesn't really exist, but they're presenting it as this. This is a weird, like combination of the two I feel like must be happening. But one, you don't really get the chance to see it, maybe because, you know, we just don't or the people don't want to talk about it, which is totally valid here. It happened to someone who is already public facing, as you say, they're already a content creator producing the sort of content that may attract harassment and trolls and that sort of thing and then they're able to actually lay it out as well. So Kylie then does go to TikTok to talk about what is happening and we'll just play a section of that now so people can hear it for themselves.
- [45:50]
Kylie Brewer
This is kind of embarrassing to talk about, but I recently found out someone has been using AI to remove my clothing and they've been posting these nude photos of me online and on an OnlyFans account. I don't have only fans. I've never consensually posted any nude photographs. I don't have any nude photos out there. I've been very careful. I have no tapes, nothing. So to realize that someone is not only creating these images without my consent, but also profiting off of them and posting them on a website that I have nothing to with. Do to do with is sickening. I truly believe this is an attempt to intimidate and silence me because of how critical I've been of men, specifically white men, as well as racists. And they have already attempted multiple doxing campaigns. They found my phone number, had to change it. Although it's not a surprise, it is still distressing. And I wanted to let you guys know, this isn't real, but for any female content creator, for anyone who has their Instagram public, this could happen to you. It probably will happen to you. You're already seeing the profoundly negative impact of AI, and that is why we absolutely need regulation. I am especially unsettled and terrified that this software is going to be used on children. If you could spread the word and interact, I'd really appreciate it. Especially because this video keeps getting taken out.
- [46:52]
Joseph
So she posts that TikTok. That's when you learn about it, right? And then you. You message her, Sam, what was that conversation? Like, what does she tell you?
- [47:03]
Sam Cole
I mean, she. She explained to me a lot of the ways that it feels to be a target of this sort of harassment, which is. It's like no one is. No two people are the same when they deal with this sort of targeted abuse imagery. But there are so many common themes that happen every single time this happens. It's like, first of all, you're getting messaged by people who you don't know in a lot of cases, or like acquaintances or people you haven't talked to in really a long time, like past schoolmates, whatever it is, people messaging you saying like, hey, heads up. Just want to let you know something weird's going on, which is just unsettling in so many ways. Like, there are layers to that. It's like, first of all, people have seen these images, so they must be spreading pretty widely. So lots of people have seen them, and they've seen them before you have even had a chance to see them in a lot of cases. Often this is how people find out that they're even in abuse imagery. Deepfakes are not AI or not. This is how they learn that they are posted on porn sites without their knowledge or consent, things like that. So is people messaging them and saying, hey, heads up. Which it's not that those people are doing anything wrong by saying that, but you're already kind of behind a ball that you didn't even know existed. Yet it's like you're already on the back foot as far as getting this stuff under control because it spreads so widely that people you don't even know have found it. So she talked about that, and she talks about that quite a bit in the piece. But she also mentioned, and this is something that I've heard from lots of different people who've been targeted by Deepaks especially, is before it happens to you, you don't really realize how it feels. And. And also the feeling that people assume that because it's not real, that it's not damaging or it's not something that you should be that stressed about, which.
- [49:03]
Joseph
Is sort of the opinion of someone. Someone is only going to say that if they haven't experienced it themselves, right?
- [49:10]
Sam Cole
Yeah, exactly. And so many people who have then experienced it say, oh, I didn't even realize that. I was like, that's really horrible. But, like, experiencing it just kind of drives home like, oh, this is actually, like, really traumatizing. And she made the point that. And I don't know if it's like, worse is not the right word. It's just like, different and weirder and kind of adds to the uncanny Valley. Effect of it is that you don't have control over these images because they're AI. So anyone can be making any images of you at any time. It's not just like there's existing content that was shot and it's being posted without your consent. It's that you don't really have any control over anything that happens anymore as far as your imagery. So, yeah, I mean, that's definitely, I think, a valid point for sure. And she also mentioned that for someone. And this is something that Cutie Cinderella Twitch Streamer talked to me about years ago when she was targeted with Deepfakes, is that as a sexual violence survivor, it just compounds all of that to a hundredfold. It makes all these things resurface. Or it can. Again, everybody's different, but it can make a lot of these old traumas resurface in a way that you don't even realize that they were still there. It's like Cutie Cinderella talked about resurfacing, like, disordered eating and body image issues and things like that. After she had seen AI images of herself in, like, sexual scenarios, and Kylie talked about something similar, she was like, you know, this is this lack of control feeling is something that really brings up a lot of traumatic things from my past. So I think just again, it's like everyone's different. Everyone experiences something Different. But it's just such a telling thing that these points hit people in the same way or in a similar way every single time. It's like the damage is very definable in a lot of these cases.
- [51:20]
Joseph
Yeah. And we've only really been able to figure that out recently because unfortunately, it takes a minute for there to be a lot of victims and then for journalists like yourself to be able to then speak to a bunch of victims across time and then see, oh, wait, there are. There are similarities in the responses here from people. Even while the technology is barreling forward, getting more and more powerful, accurate, sophisticated, however you want to characterize it, easier to use all of that sort of thing. And then you get, of course, more and more of these reactions as well on that. Lack of control. Yes, lack of control of being able to make the images and crucially, the distribution, because somebody has made them and now they've made an OnlyFans. And it's not her OnlyFans. You know, she didn't create it. She can't immediately shut it down. So she doesn't control that. What did OnlyFans do here? Exactly.
- [52:20]
Sam Cole
So by the time she checked these DMs and clicked on the links that people sent her, the account was down. OnlyFans doesn't allow impersonation accounts or deepfakes. It's in their terms. So I'm sure her fans or followers or people who knew her probably reported the account and it came down pretty quickly, it seems. It's still. It's a huge problem that it was up in the first place. It's not good that it existed ever. I think this is something that OnlyFans needs to figure out quickly. How people can open accounts and then change the names or profile pictures or the content of the account without the oversight, apparently. I don't know what's going on there. I ask OnlyFans so many questions for every single story I do, and they never, ever reply to me or they haven't in years. So I don't know what happened in this case.
- [53:12]
Joseph
They just don't respond, though.
- [53:14]
Sam Cole
They haven't replied in a really long time to anything I've sent them. So it just kind of goes into a void. They used to. They used to have like a PR team and they used to reply, but I don't know, not anymore. So, yeah, we don't really know in this specific case and they probably wouldn't comment on a specific case anyway. But it's not good that it was ever online in the first place, obviously.
- [53:36]
Joseph
Well, and there are other platforms as well. I won't name them because I don't know their exact policy off the top of my head. And there's no evidence they were being used in this case. But there are other platforms like OnlyFans where somebody might just go and make an account as well. If OnlyFans is shutting it down for one of its.
- [53:52]
Sam Cole
I mean, this is something that we see people doing all the time with AI generated sexual imagery, is spinning it out and trying to profit off of it. I mean, I wrote that piece about the, the guy with the Ray Bans walking into massage parlors and then trying to sell or like, you know, who knows what he was actually selling, but selling that content on subscription sites and people were doing it on Patreon. As far as like creating AI generated sexualized images and then putting them on Patreon and putting them behind a paywall, which sucks. It's like you're monetizing at that point the abuse of someone else under their name. It's just, it's so. It's a fucked up ecosystem that I expect better from OnlyFans as far as preventing this stuff. But there are other channels that this happens in as well.
- [54:45]
Joseph
Yeah, as you say, at least they closed it down seemingly before she could access it and that sort of thing. But it still exists in the first place. To wrap up, what are the legal ramifications here, if any, for somebody who took a likeness, generated those images, then created a fraudulent OnlyFans, is there potential legal ramification? I'm not expecting anything to happen. I'm just worried. I'm just curious about the legality.
- [55:14]
Sam Cole
Yeah, so a lot of it is the way that it stood for a really long time. Lots of states now have deepfake laws on the books. Like if you're in a deep fake, you have legal recourse. But in a lot of states, revenge, like quote unquote, revenge porn statutes have existed for a really long time and they're just hard to enforce. It's like you're dealing with the Internet, you're dealing with anonymous users a lot of the time, so it's hard to find the person to sue them in the first place, but it's possible. And that is something that victims have tried. It's also, it's hard to go to the police about this stuff because police don't really know what's going on with AI still, even though it's, you know, 2026 at this point, they going to the police about gendered violence and sexual Violence in general is really hard and often re traumatizing. We have the Take It down act, which is the first federal level deepfakes law, but it has a lot of its own problems. Created a 48 hour turnaround time for platforms to get deepfakes taken down when they're reported. But Trump has said, yeah, I'm going to use this law to take down the stuff that is mean to me. And Melania Trump and Ted Cruz have been the champions of that law. So we don't really know what's going to play out with that yet. But it's relatively new. AOC and Paris Hilton are pushing it's. They did a press conference a couple weeks ago about this law. It's called the Defiance Act. I love the acronyms that they think of for these things.
- [56:53]
Joseph
Somebody, somebody in the Congressional office is tasked with coming up with those. And I love it every time.
- [56:58]
Sam Cole
And you know that they know that they ate with this one. It's the Disrupt Explicit Forged images and Non Consensual Edits act like, okay, but that just passed the Senate. It's going to go to the House. But it would open up a venue or avenue for people who've been the target of deepfakes to sue the people making the content, which is different from some of the other laws which allow targets to sue or go after platforms. So I think that's a decent step. All of these are decent steps, but none of them are encompassing the entire issue yet without creating more censorship and more problems for people producing consensual adult content in the process. So that's where that's at. We'll be following that and see where that goes. But yeah, it's hard. Legally, it's still incredibly hard.
- [57:58]
Joseph
Yeah, we'll be following in and depending on the case, you can even fall back to an old cyber stalking charge. I don't think that would probably apply here, but you sometimes see that, right? And you can fall back on those sorts of things. We'll leave that there. If you are listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about how cops are buying GeoSpy, an AI that can allegedly geolocate photos in seconds. You can subscribe and gain access to that content at 404 Media co. As a reminder, 404 Media is Joomlist, founded and supported by subscribers. If you do wish to subscribe to 404 Media and directly support our work, please go to 404 Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the Subscribers Only session section where we talk about a bonus story each week. This podcast is made in partnership with Kaleidoscope and Alyssa Midcalf. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really does help us out. This has been 404 Media. We'll see you again next week.
- [59:20]
Emmanuel Mayberg
Foreign.
- [59:26]
Ad/Promo Voice
It's tax season, and at LifeLock we know you're tired of numbers, but here's a big one you need to hear. Billions. That's the amount of money and refunds the IRS has flagged for possible identity fraud. Now here's another big number. 100 million.
- [59:40]
Emmanuel Mayberg
That's how many data points LifeLock monitors every second. If your identity is stolen, will fix it.
- [59:46]
Ad/Promo Voice
Guarantee one last big number. Save up to 40% your first year. Visit lifelock.com podcast for the threats you can't control. Terms apply.