On with Kara Swisher: Deepfake abuse panel, featuring Laurie Segall

Summary

Podcast Summary: "On with Kara Swisher" – Deepfake Abuse Panel

Release Date: July 16, 2026
Podcast: Mostly Human with Laurie Segall (shared episode from "On with Kara Swisher")
Title: Deepfake Abuse Panel, featuring Laurie Segall
Host: Kara Swisher
Guests:

• Carrie Goldberg (victims’ rights attorney)
• Laurie Segall (tech journalist, CEO of Mostly Human Media)
• VS Subramanian (Computer Science Professor, Northwestern University)

---

Episode Overview

This panel discussion explores the rising crisis of non-consensual deepfake pornography—AI-generated sexually explicit videos and images that use a person’s likeness without their consent. What began as an issue impacting celebrities has now spread to everyday people, including teens. The episode unpacks the technology’s rapid evolution, legal grey zones, failures of major platforms, ongoing harms to victims, and paths forward in regulation and accountability.

---

Key Discussion Points & Insights

1. Scope and Pervasiveness of Deepfake Abuse

[09:29–12:53]

• Kara Swisher introduces the topic and asks guests to describe the current landscape.
• Carrie Goldberg: Deepfakes initially targeted celebrities, but now affect even middle schoolers. The landscape changed significantly with the ease of creation and dissemination on platforms like X (formerly Twitter).
  • "Everything totally changed when Elon started sending around, you know, pictures of toasters in bikinis… everyone was able to create deepfakes and then immediately publish them on the X platform." [10:20]
• Laurie Segall: The abuse migrated from celebrities/politicians to regular girls and women; now rampant among teens. Normalization among youth is alarming.
• VS Subramanian: The technology is so advanced that even experts struggle to tell real from fake; both human and AI detection are needed.
  • "It's really gotten to a stage where even experts … have difficulty eyeballing something, listening to something, and telling if it's real or fake." [11:29]

2. Evolution & Ease of Creating Deepfakes

[12:15–12:55]

• VS Subramanian: Major improvement is quality—edges, face-swapping, and seamless merges are now easily doable for anyone.
  • "Today... much of that has been worked out and ease of use for normal people to do so, Correct?" – "Indeed. Absolutely." [12:50–12:53]

3. Case Study: Mr. Deepfakes Investigation

[12:55–15:41; 60:01–62:37]

• Lori Segall details her investigation:
  • Mr. Deepfakes was the largest deepfake porn site (“17 million visits per month at its peak.”)
  • Hyper-explicit, violent fakes of women—victims often had no recourse as the site creator was anonymous and laws hadn’t caught up.
  • Community forums normalized and monetized abuse.
  • Investigators tracked down the owner; exposure and legal friction, combined with public pressure and technical interventions (like server removals), eventually led to the site shutting down.
  • "He had been able to create a website that enabled a whole new wave of deepfake abuse because ... when Reddit banned sexually explicit deepfakes, we were able to track where his username was. Talking about seeing an opportunity that could be profitable..." [37:08]
  • Despite being exposed and losing his job, the perpetrator faced little legal consequence.

4. Victims' Experiences & Harms

[15:41–18:39; 21:37–24:13]

• Joanne Chu (artist, actor, and victim): Discovering her face used in deepfake porn was “akin to physical assault.”
  • "It's very much akin to physical assault… it doesn't matter if you're famous or not… it's like they have a whole alphabetized database." [15:52]
• Goldberg & Segall:
  • The psychological impact is profound—victims report shame, humiliation, loss of agency, and even suicidal ideation.
  • Example: A mother found out her entire neighborhood’s faces had been deepfaked by a friend.
  • "Actually this causes deep, deep psychological abuse and it can be used to try to digitally abuse a woman. I say a woman, anyone." [13:08]

5. Role of Platforms & Technology Infrastructure

[31:40–36:29]

• VS Subramanian: Deepfake generators rely heavily on scalable, affordable tech—cloud infrastructure, GPU servers, APIs, etc. Platforms play a key role in enabling or shutting down abuse.
  • Tech can’t catch up: "Small changes can make a big difference… what constitutes an act that's sexual in nature can vary a lot."
  • "Those resources are heavily supported by cloud infrastructure providers… and so part of the onus is placed on them." [32:17]
• API access allows at-scale abuse: Not just individuals but coders automating massive deepfake creation.

6. Monetization & Community Incentives

[36:29–39:36]

• Segall: Mr. Deepfakes became an “ecosystem”—users paid for custom fakes. Payment platforms, lack of laws, and forum incentives fueled proliferation.
  • "People were probably making more money than the guy behind Mr. Deepfakes off of this platform… once Google deranked the platform, there were all sorts of small steps that actually created a friction..." [37:08]

7. Legal Battles & Regulatory Responses

[39:36–41:55; 24:38–27:46; 48:28–52:07]

• Section 230: Shields platforms from responsibility for user content; Goldberg argues for product liability battles—noting AI tools themselves produce the illegal images, not just host them.
  • "We're suing them not for publishing the content, but for basically harming people through its defective design..." [40:07]
• Take It Down Act & Defiance Act: New regulations require platforms to remove non-consensual deepfakes within 48 hours, but enforcement is weak; victims can't sue platforms directly.
  • "The problem is… the victims cannot enforce them. And so when we're talking about who's the bad guy… the worst guys are the platforms." [51:15]

8. Enforcement, Workarounds & International Challenges

[41:55–44:44; 63:55–64:43]

• Perpetrators use technological workarounds (e.g., stable diffusion, API tricks, off-platform computing) to evade detection and platform guardrails.
• VS Subramanian advocates for coordinated international legal frameworks, possibly under organizations like INTERPOL; domestic laws alone fall short.

9. Education, Culture, and Prevention

[52:14–57:18]

• Panelists stress the need for school-based and parent education on the risks, ethics, and consequences of deepfake use among youth.
  • Segall: Culture is shifting—the presence of federal law sends a strong message, but real change requires both accountability for platforms and increased awareness in communities.
  • Goldberg: Massively reduced image-based abuse happened after criminalizing revenge porn; similar laws for deepfakes could help, but must include holding platforms responsible.

10. Paths Forward & Hopes for the Next Five Years

[62:37–65:58]

• Goldberg: Wants strong penalties and financial compensation for victims; platforms and supporting app stores should be held accountable.
• VS Subramanian: Criminal law must be international, not just US-based, given the global nature of distribution.
• Segall: Hopes for major cultural change in both tech and general culture—wants AI safety, empathy-driven education, and industry accountability.
  • "My gripe with Silicon Valley… is they're sitting there talking about AGI… Meanwhile here on earth… we have children who are ending their lives because of deepfake sextortion, and people are sleeping on it." [64:50]

---

Notable Quotes & Memorable Moments

• Kara Swisher: "When I started my career, pornography was at the heart of the problems with the early Internet... the abuse of it was built into the system at the start. A lot of people were aware of it then and have had a very hard time dealing with it, even as technology has gotten better and better at being more and more abusive." [02:23]

• Joanne Chu (victim): "It's very much akin to physical assault… it doesn't matter if you're famous or not." [15:52]

• Lori Segall:

  • "You never wanna be the person talking about deep fake porn at the dinner table, but turns out I was... people were like, well, the images aren't real, so it doesn't matter." [13:08]
  • On her own victimization by deepfakes: "I felt shame, I felt humiliation. I felt like the world had seen me naked, to be quite honest with you… Imagine what that's gonna be like for an 11 year old girl." [21:37]

• Carrie Goldberg: "There's just no reasonable safe use of this product but to harass people publicly." [24:38]

• VS Subramanian: "You have a bigger audience of bad guys. Or as Kerry mentioned, just teenagers who don't know better and haven't been taught better." [20:27]

• Panel consensus on regulation:

  • "Until we have laws that victims can enforce themselves… it's not really harming the true offenders." — Goldberg [53:29]

---

Timestamps for Key Segments

• [09:29] – Start of main panel discussion
• [10:04] – Pervasiveness and evolution of deepfakes
• [12:55] – Segall’s investigation of Mr. Deepfakes
• [15:52] – Victim testimony (Joanne Chu)
• [21:37] – Personal impact stories and victim harm
• [24:38] – Discussion of legal actions against XAI, Grok, and platform liability
• [31:40] – Technical infrastructures, APIs, and scaling
• [36:29] – Monetization and community incentives
• [39:36] – Law and regulation, Section 230, product liability approaches
• [48:28] – Defiance Act and current legal landscape
• [52:14] – Lightning round: expert question about "Take It Down Act" implications for teens
• [57:38] – How to request take-downs as a victim
• [58:26] – Balancing regulation with positive uses of generative AI
• [60:01] – What brought down Mr. Deepfakes
• [62:37] – Hopes and predictions for next five years

---

Action Steps / What Can Individuals Do?

• Platforms are required to offer easy reporting and takedown flows for deepfake victims (per “Take It Down” Act). Use the reporting tools available on social platforms and document all harassing images/content.
• Public pressure, investigation, and education are critical in pushing both legal and cultural progress.

---

Conclusion

Deepfake abuse has rapidly become a mainstream threat, with widespread psychological and reputational harms disproportionately impacting women and teenagers. Although some laws now exist, ongoing legal loopholes, technological workarounds, and a lack of platform accountability hinder real progress. Panelists call for stronger, victim-centered regulation, greater industry self-policing, cross-border law enforcement, and robust public education to combat future waves of abuse.

---

Host’s Final Words:
“We’re going to keep coming back to it again and again and hopefully see something happen by raising awareness of it. For sure.” — Kara Swisher [66:07]

---

(Summary compiled by reviewing the full guest discussion content. Ads and unrelated promos have been omitted.)

Transcript

[Kara Swisher]

This is an iHeart podcast. Guaranteed Human.

[iHeart Podcast Announcer]

Run a business and not thinking about podcasting, think again. More Americans listen to podcasts than ad supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. Learn how podcasting can help your business. Call 844-844-IHeart.

[Ben Nadifray]

Thirty years ago, an island off Manhattan almost brought down New York City.

[Narrator (Revisionist History Teaser)]

Who will you trust? Your friends and neighbors? Or you trust the people five miles overseas?

[Ben Nadifray]

This is a story about neighbors turning on each other and what happens when a forgotten place decides it's had enough.

[V S Subramanian]

But we're not stopping, are we?

[Ben Nadifray]

Listen to revisionist history, the Staten island problem on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. 150 years ago, they were hunting us down to kill us.

[Unnamed Historian/Commentator]

And now they're hunting down immigrants to deport them.

[Rebecca Nagle]

This is First America, the true story of how the United States came to be and how we got to this present moment. Listen to First America on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Mangesha Tigadar]

I'm Mangesha Tigadar and I'm back with a new season of my podcast, Skyline Drive. This time I talk to scientists, biopunks, curmudgeons, blue zoners, super seniors, and Goa's top cryotherapy lab to try to understand this obsession with living forever and what it means for all of us. And I get into a bit of trouble along the way.

[Kara Swisher]

I'd say probably start bone smashing. That doesn't work. To make it look more defined. They say it works.

[Rebecca Nagle]

I don't know.

[Mangesha Tigadar]

Listen to Skyline Drive, how to live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Lori Siegel]

Hi, mostly human listeners, it's Lori Siegel. Today I have a treat for you. I'm sharing an episode of Kara Swisher's podcast On with Kara Swish. Cara hosts a panel of deepfake experts, including me, to discuss the threat, the current scope and the future of sexually explicit deepfake abuse. Now, if you've been listening to our four part series, Searching for Mr. Deepfakes, this is a perfect cap. If you're new to the subject, it's a fantastic primer. Hope you enjoy.

[Kara Swisher]

Hey folks, a word of warning before we begin. Today's episode discusses sexual violence, including rape, graphic sexual content, and suicide. So you might want to keep this in mind when you decide if and where to listen. Hi everyone from New York Magazine and The Vox Media Podcast network. This is on with Kara Swisher. And I'm Kara Swisher. My guests today are attorney Carrie Goldberg, tech journalist and CEO of Mostly Human Media, Lori Siegel and VS Senior Subramanian, the Walter P. Murphy professor of Computer Science at Northwestern University. We're talking about non consensual deep fake pornography, or AI generated explicit videos and images that use real people's likenesses without their consent. Celebrities were often the target of early pornographic deep fakes. Now according to one survey, one in eight teens personally knows someone who has been targeted. Nearly all deepfake porn depicts women. Despite laws and moderation efforts, pornographic deepfake websites and so called new nudify apps continue to multiply. And a recent analysis from Wired found that Elon Musk's Grok chatbot is still being used to generate and host non consensual deep fakes. I am not surprised in the least. Carrie Goldberg is a victims rights attorney. Her law firm specializes in fighting Internet abuse and revenge porn. Journalist Lori Siegel recently released Searching for Mr. Deepfakes, an investigative docu series created for TikTok in partnership with Paris Hilton. It follows an investigation to unmask the anonymous operator behind one of the Internet's most notorious deepfake porn platforms. V S Subramanian is a top computer scientist and deepfake expert. He led the team that developed the Global Online deepfake Detection System, a free platform to help journalists detect deepfakes. I think it's critically important to talk about this issue. When I started my career, pornography was at the heart of the problems with the early Internet and the uses of it. And the abuse of it was built into the system at the start. A lot of people were aware of it then and have had a very hard time dealing with it, even as technology has gotten better and better at being more and more abusive. So it's critical that we keep talking about it, we criminalize this and we start to hold platforms, big platforms, responsible for the stuff that they distribute. Our expert question today comes from Wall Street Journal tech reporter Georgia Wells. This is an important conversation and one we will keep having, so stick around. And one more thing before we get started, if you're going to in Washington, D.C. on July 16, please join us live at the Johns Hopkins University Bloomberg center for a taping of On. I'll be talking to Gina Raimondo, Commerce Secretary under President Biden and former Rhode Island Governor, about what AI means for the American workforce. And as a special bonus before that conversation, I'll be speaking with Johns Hopkins University President Ron Daniels and the University of Notre Dame president Father Robert Dowd about how universities are approaching AI and workforce issues. You can get tickets and learn more [e-mail] Karaswisher Live.

[iHeart Podcast Announcer]

Run a business and not thinking about podcasting? Think again. More Americans listen to podcasts than ad supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. So whatever your customers listen to, they'll hear your message. Plus, only iHeart can extend your message to audiences across broadcast radio. Think podcasting can help your business? Think iHeartrade, iHeart streaming radio and podcasting. Let us show [e-mail] that's iheartadvertising.com

[Joanne Chu]

the

[Unnamed Historian/Commentator]

declaration, which is full of these beautifully rendered, you know, sentences and paragraphs about enlightenment ideals, does also have this darker history to it.

[Rebecca Nagle]

Why is it important for the darker part of the Declaration of Independence and the American Revolution? Why is it important that Americans know about it?

[Unnamed Historian/Commentator]

Well, if we don't understand the full context in which our nation was founded, we won't understand the full context in which our nation now finds itself.

[Rebecca Nagle]

I'm Rebecca Nagle. Gohin Tawadon Jalecayetli Gay la citizen of Cherokee Nation.

[Lori Siegel]

Are you guys big Chiefs fans?

[V S Subramanian]

Hell yeah.

[Rebecca Nagle]

This is First America, the true story of how the United States came to be and how we got to this present moment. Listen to First America on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Ben Nadifray]

Thirty years ago, an island off Manhattan almost brought down New York City.

[Narrator (Revisionist History Teaser)]

Who will you trust? Your friends and neighbors? Who will you trust? The people five miles overseas.

[Ben Nadifray]

This is a story about neighbors turning on each other and what happens when a forgotten place decides it's had enough.

[V S Subramanian]

We reached for a feeling that exists in this city. But we're not stopping, are we?

[Ben Nadifray]

I'm Ben Nadifray, and for my new series on revisionist history, I took a deep dive into Staten Island's attempt to secede from New York city in the 1990s and what it all has to do with the politics of resentment that dominate America today. Listen to Revisionist the Staten island problem on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Mangesha Tigadar]

I'm Mangesha together and I'm back with a new season of the podcast Skyline Drive. This time I'm diving into a rabbit hole of peptides, organoids, blood boys, blue zones and brain repl placement to Try to understand what this longevity obsession is all about and what it really means to live forever for all of us. I learned about some rad science.

[Joanne Chu]

I can make a brain for you and then we can test what drug is the best for your brain as opposed to his brain.

[Mangesha Tigadar]

Here are some hard truths.

[V S Subramanian]

I would expect Indians to age faster, but I did not expect it to be almost a four to five year

[Mangesha Tigadar]

acceleration and get myself into a world of trouble.

[Kara Swisher]

I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works. I don't know.

[Mangesha Tigadar]

Listen to Skyline Drive, how to live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[iHeart Podcast Announcer]

It is over.

[Kara Swisher]

Kerry, Laurie and Vs, thanks for coming on on.

[Lori Siegel]

Good to be here.

[V S Subramanian]

Thrilled to be here.

[Carrie Goldberg]

Hello.

[iHeart Podcast Announcer]

Hello.

[Kara Swisher]

So I talked to experts back in January about the proliferation of sexualized non consensual AI deepfakes, including images of children on Elon Musk's X that were generated by its Grok chatbot. Obviously, AI created deepfake porn hasn't gone away on Grok or elsewhere and the technology continues to improve. I'd like each of you to explain the current scope of the problem and how pervasive it is and who it' affecting and how technically advanced it is. We're going to get more into detail, so go high level here. Carrie, we'll start with you, then Lori, then Vs.

[Carrie Goldberg]

So I'm a plaintiff's attorney that sues tech companies and generally I see problems at the very early stages. So, you know, back probably seven years ago, we started getting situations where celebrities and really high level content creators were being deepfaked. And then over the last just, I'd say year or so, it started trickling down to middle schoolers getting deep faked. But all the apps involved were these pretty obscure things. But then everything totally changed when Elon started sending around, you know, pictures of toasters in bikinis. And it just unleashed this never before seen sort of situation where everyone was able to create deepfakes and then immediately publish them on the X platform.

[Kara Swisher]

So that's where you saw it?

[Unnamed Historian/Commentator]

Lori?

[Lori Siegel]

Yeah. When we started looking at this years ago, it was very much focused on celebrities, oftentimes women in power, politicians. It seemed like the first line of defense. And I remember thinking at the time, oh no, you know, you might not care about these people, we should care about these people. But what this means is this is gonna happen in our high schools. And unfortunately that's what we're seeing happening So I would just say it's rampant and it's happening in our communities. It's happening to women, to all sorts of folks. And unfortunately, it's something that teenagers are learning and it's becoming commonplace.

[Rebecca Nagle]

Go ahead.

[V S Subramanian]

Vs. KARA I'm a technologist, and so I've spent a lot more time thinking about the technology. Over the last five, six years ago, the ability to create deep fakes was in its infancy. But over the last five years, and in particular over the last year or two, it's really gotten to a stage where even experts like me on the detection of deepfakes have difficulty eyeballing something, listening to something, and telling if it's real or fake. So what we are seeing is something where you really need humans and AI to come together, to work as some kind of partnership to try and figure this out. And until we can get that right, I don't see an end in sight.

[Kara Swisher]

And how has the technology improved in the last year? Just ease of use.

[V S Subramanian]

Ease of use has always been there. The improvement has been in the quality of the generator deep fakes. So today, for example, we're seeing things like face swapping. So I could record something of myself saying something thoroughly objectionable and then put, you know, replace my face with that of a celebrity. In the past, that was difficult to do. There were things around the edges of the face that would be noticeable. Today, much of that has been worked

[Kara Swisher]

out and ease of use for normal people to do so, Correct?

[V S Subramanian]

Indeed. Absolutely. Yeah.

[Kara Swisher]

So Mr. Deepfakes launched in 2018 and was once again the largest deep fake porn site online, getting 17 million visits a month at its peak. Lori, talk about your investigation into Mr. Deepfakes and how it began.

[Lori Siegel]

It began a bit with an obsession, to be quite honest with you. I remember it was 2022, ChatGPT had launched. Everybody was talking about how AI was gonna just do these incredible things. It's gonna cure cancer. It's going to, you know, give us all this free time. And of course, I mean, Kara, you know this better than anyone. We're all kind of looking at this and saying, okay, there are a lot of unintended consequences. And this is moving very quickly without the correct guardrails. And someone told me about Mr. Deepfakes. I will never forget going to this site and opening it up and seeing thousands, thousands of highly sexually graphic images, deepfakes of women who did not consent and they were violent in nature. It was a website where a creator could say, you know, could go and make money and someone could go and say, I want this woman doing this. And they could pay for it. So it was beyond just a website, it was a platform. And then what was even more concerning as we were digging into it was, you know, there was everything from training data of women's faces to create hyper realistic deepfake pornography. And then there were forums where people would exchange notes on, here's the latest deepfake tools, here's how you can do this. You can pay a premium and get a deepfake training manual. And so my initial thought was this isn't gonna just be for these high profile women. And high profile, I would say let's take it with a grain of salt. And I remember thinking at the time, well, it's not gonna stop here, you know, this is just the beginning of it. And by the way, you're looking at these forums and you're seeing people say, is it wrong that I wanna do this to my sister in law? And oh, this is so easy to do. And so it's almost creating entertainment around deep fake abuse. And so as I was looking at this, I thought, well, surely there's something these women can do to help themselves. And the reality was absolutely not. There was nothing they could do because the man behind it was anonymous and he'd been anonymous for six years. And even so, even if he wasn't anonymous, the laws hadn't caught up. And even when I went out and started talking about this, and by the way, you never wanna be the person talking about deep fake porn at the dinner table, but turns out I was, people were like, well, the images aren't real, so it doesn't matter. And I'm sure everybody on here is kind of just shaking their head. So. We also had a cultural problem. We needed people to realize that these are so hyperrealistic. Actually this causes deep, deep psychological abuse and it can be used to try to digitally abuse a woman. I say a woman, anyone. And so that was what was alarming. And so we decided we had to go find him and raise awareness. So that was the beginning of the investigation.

[Kara Swisher]

Let's hear from Joanne Chu. She's an artist and actor, one of the women featured in Laurie's investigation. She's talking about the moment when she realized her image had been used in porn on Mr. Deepfakes.

[Joanne Chu]

I was just bored one evening and I'm just gonna google my name and see what comes up. I kept scrolling down and then I said, what? What is this? It's like my face on very disturbing and graphic content. It's you Know Joanne Chu or Joanne Chu. In the beginning it was just a few listings. Mr. Deepfakes came up social media girls, E. Porner. And then when I typed it in again, it multiplied. It's very much akin to physical assault. I have like less than 3,000 followers. But it doesn't matter if you're famous or you're not just looking at Mr. Deepfakes. It's like they have a whole alphabetized database. At one point when I typed in my name, my stuff wasn't coming up first anymore. It was all of this content.

[Kara Swisher]

So she compares it to being physically assaulted. Kerry, does this reflect what you've heard from your clients? Because it sounds like perpetrators are telling themselves that it isn't real harm.

[Carrie Goldberg]

Yeah, I remember this when I first started my firm back in 2014 and it was everyone's attitude about revenge porn and how, you know, this is just an online problem. And back then there were all these revenge porn websites and people were searchable through Google which would lead people to the search engine results and it would just be pages and pages of nude images. Like all the harm was really, really underestimated back then. And it took a huge like amount of advocacy to change the public opinion about the shame around being, you know, nude online. And we're just, we're seeing something really, really similar. The difference though is that back, you know, 10 years ago, women were being blamed for having shared an intimate image in the first place. And with Deepfakes, I don't see that same kind of victim blaming because here no one, no one consented to the image in the first place. But there is still a lot of, I think marginalization and minimizing of the harm.

[Kara Swisher]

That it's not physical assault, that it's,

[Carrie Goldberg]

yeah, I feel it is conceptually different than a physical assault. But that doesn't mean that it's not still really, really harmful and humiliating for victims. Especially like I see so many young people who are like victims and it's other young people's doing it to them. You know like where it's peer on peer in junior high school and high school settings. And a lot of times the offenders just think that they're doing a prank or if you know. And always the defense is, well I barely shared it with anybody, I didn't post it online. But these things find a way to get published.

[Kara Swisher]

So deepfake technology first emerged around 2015. VS Talk about the online communities that exist for sharing information, making so called nudify tools accessible and how has it evolved over the past 10 years in terms of how it's used.

[V S Subramanian]

So, Kara, there are a number of nudification AI tools. So these are tools that are of various varieties. One, as I mentioned earlier, is face swapping, where you've got an individual who's nude performing some sex act and their face has been replaced with the face of somebody like Ms. Chu, who you alluded to a couple of minutes back. So that's one class of techniques. Another is to use massive databases of training data of existing nude individuals to create and generate new nudes of people who are hyper realistic, who are performing acts which the real individual portrayed never performed. So that's another class of techniques. And I don't want to go into the gory details of how that's done, but there are a number of techniques. These techniques have evolved better and better every single year. And so as a consequence, what we see is really a huge number of websites that can disseminate such stuff. Not just the produced content, but also the tools that are used to create and disseminate this stuff. And that's typically done on the Dark Web through some kind of distribution network on the Dark Web.

[Kara Swisher]

And what changed when Elon made it easy to do so? Because for all eyes in the Dark Web, you have to actually make an attempt to go get it, right? It's like going to a porn store or going to a porn movie or something back in the old days. Talk a little bit about how that changed.

[V S Subramanian]

Well, I can't speak specifically to Elon and what he did, but what I can say is that the more these tools are available through normal channels, the more the audience that gets access to them. And as soon as you have a big audience, you have a bigger audience of bad guys. Or as Kerry mentioned, just teenagers who don't know better and haven't been taught better, possibly because their teachers and their parents, however decent they may be, were not aware of the threat. And by the time they figured out, it was too late.

[Kara Swisher]

Deep fakes and nudify apps are also a big problem in schools. As we mentioned earlier, a recent survey of roughly 550 US teens found that over half had created at least one image using notification tools, while a third of the teens reported having their image created and shared non consensually. I'm sure the numbers are higher, actually. Sites like Mr. Deepfakes and Apps like Grok have, as we noted, have normalized creating non consensual, in some cases violent sexual imagery, often of women. Lori, talk about the consequences for young people.

[Lori Siegel]

Yeah, you know, I'LL start. I always like to say, like to try to make it a little bit personal. I'll never forget. And I'm gonna take it here and I promise I'll get to young folks, but I'll never forget. I think it was 20, 23 when a group of technologists approached me and said, can we have AI attack you and see what it could find? And because I am a crazy person, I said, sure, let's see. And what they did was they had AI come up with an attack against me. And it took some true things, right? Like I've interviewed Mark Zuckerberg before or I'm a technology journalist. And then it ended up in front of an audience, it ended up creating, they created sexually explicit deepfakes of me. And I'll never forget, because this. I had consented. I didn't realize they were gonna do that, but I had consented to this demo. And I'll never forget looking at the audience, look at these sexually explicit deepfakes of me. And I could, as a person who generally forms sentences for a living, I stopped being able to form sentences. I felt shame, I felt humiliation. I felt like the world had seen me naked, to be quite honest with you. And then I thought to myself, well, if that's how I feel and I know what this technology is, imagine what that's gonna be like for an 11 year old girl, you know, And I think that's what we started hearing. So I've spoken to a lot of parents, a lot of teenagers who've had this happen to them. I spoke to one woman who said, and you know, she's in her early 20s, she said, she after this was happening and she went out and she tried to get this stopped. And people said, there's nothing you can do. They blamed her. Very similar to what Carrie is talking about. Back when we were talking about nonconsen pornography, people didn't quite understand it. She told me she walked to the roof of her building and she considered jumping off. And you talk to a politician who describes this as digital rape. And so for me it was really confusing because I was looking at this website that for me this website represented this dystopian era where we lose consent in an AI era where our most intimate qualities can be weaponized against us. And then externally we were hearing, but it's not real. It doesn't matter when it really impacts the way a young person, any person, walks through the world. We spoke to a woman named Molly, who was a mother who had this, her husband's best Friend deep faked the whole neighborhood, 80 women in the community, and created sexually explicit deep fakes. And she said it impacted how she would go on the bus and look at people and wonder if they'd seen this. And she had to go on a, she said a porn hunt for her own face on the web. And I think it really impacts from a human perspective how you walk through the world and how you hold yourself and it can have reputational harm.

[Kara Swisher]

Right. So Kerry, your law firm and your word sues, quote, psychos, pervs, trolls and toxic tech. In January, your firm filed a lawsuit against X AI on behalf of Ashley St. Clair, a conservative influencer who had a child with Elon Musk. She alleges that his company's chatbot was used to create and disseminate sexually explicit, nonconforming, consensual images of her. Talk about the lawsuit and how is it representative of bigger legal issues around

[Carrie Goldberg]

deepfake porn when Grok suddenly had this new capability of being able to create nudification images? This was the first time in history that a nudifying tool was integrated into a widely used social media platform, which combined the ability to just generate an image with the widespread dissemination network. And so suddenly an image could just be viewable by millions and millions of people. But also, I mean, X is for people 13 and up. And so children were also then consumers of the images, which is illegal. And there were children that were the victims of it. And the images posted were not just people in bikinis, but GROK would pose them in sexual positions, it would respond to prompts telling them to drench their body in blood and semen like fluids, hold suggestive props and just really vulgar and explicit images. And it seemed very clear that this was a, an intended rollout of a product that was known to be harmful. They were like something like 1.8 million posts of women and children being deepfaked, which accounted for like 41% of all the images posted during this nine day period. So Ashley St. Clair, as somebody who is really demonized on the X platform because of some of the hostile things that Elon Musk has said about her, was a huge target of the deepfakes. And there were just so many images that Grok generated and published on her own page. And so we sued them. We first got a, attempted to get a temporary restraining order to stop Grok from deep faking her and then we sued them for defective design and other sort of product liability types of lawsuits, negligence. And the claim that I like the best is that we sue them for public nuisance. There's just no reasonable safe use of this product but to harass people publicly.

[Kara Swisher]

So that's where it stands now.

[Carrie Goldberg]

And, well, we had to give XAI statutory notice that we were filing for a temporary restraining order. And they immediately sued Ashley in Texas, claiming that she had breached the XAI terms of service by even threatening to sue them in her home state of New York. So now we are litigating the issue in two different venues. They're also trying to use the X terms of service to get the XAI case transferred to New York. And the judge initially, last week said that XAI can rely on the X terms of service to transfer the case to Texas.

[Kara Swisher]

You mean to Texas?

[Carrie Goldberg]

Yeah. So even though X is not a defendant in our case, only XAI is. The judge still let them borrow from a different Elon company, which I think is really scary.

[Kara Swisher]

So is that where it's going?

[Carrie Goldberg]

Well, we filed an emergency petition for mandamus to try to get the stay held up, but I don't know. We don't have the outcome on that.

[Kara Swisher]

We'll be back in a minute.

[iHeart Podcast Announcer]

Run a business and not thinking about podcasting. Think again. More Americans listen to podcasts than ad supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. So whatever your customers listen to, they'll hear your message. Plus, only iHeart can extend your message to audiences across broadcast radio. Think podcasting can help your business. Think iHeart streaming radio and podcast. Let us show [e-mail] that's iheartadvertising.com

[Unnamed Historian/Commentator]

the declaration, which is full of these beautifully rendered, you know, sentences and paragraphs about enlightenment ideals, does also have this darker history to it.

[Rebecca Nagle]

Why is it important for the darker part of the Declaration of Independence in the American Revolution? Why is it important that Americans know about it?

[Unnamed Historian/Commentator]

Well, if we don't understand the full context in which our nation was founded, we won't understand the full context in which our nation now finds itself.

[Rebecca Nagle]

I'm Rebecca Nagle. Gohin Dawadon Jalecayetli Gay la citizen of Cherokee Nation.

[Lori Siegel]

Are you guys big Chiefs fans?

[V S Subramanian]

Hell, yeah.

[Rebecca Nagle]

This is First America, the true story of how the United States came to be and how we got to this present moment. Listen to First America on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Ben Nadifray]

Thirty years ago, an island off Manhattan almost brought down New York City.

[Narrator (Revisionist History Teaser)]

Who will you Trust your friends and neighbors. Who will you trust? The people five miles overseas.

[Ben Nadifray]

This is a story about neighbors turning on each other and what happens when a forgotten place decides it's had enough.

[V S Subramanian]

We reached for a feeling that exists in this city. But we're not stopping, are we?

[Ben Nadifray]

I'm Ben Nadif Haffrey and for my new series on revisionist history, I took a deep dive into Staten Island's attempt to secede from New York city in the 1990s and what it all has to do with the politics of resentment that dominate America today. Listen to Revisionist the Staten island problem on the iHeartRadio app, Apple Podcasts or wherever you get your podcasts.

[Mangesha Tigadar]

I'm Mangesha together and I'm back with a new season of the podcast Skyline Drive. This time I'm diving into a rabbit hole of peptides, organoids, blood boys, blue zones and brain replacement. To try to understand what this longevity obsession is all about and what it really means to live forever for all of us. I learned about some rad science.

[Joanne Chu]

I can make a brain for you and then we can test what drug is the best for your brain as opposed to his brain.

[Mangesha Tigadar]

Here are some hard truths.

[V S Subramanian]

I would expect Indians to age faster, but I did not expect it to be almost a four to five year

[Mangesha Tigadar]

acceleration and get myself into a world of trouble.

[Kara Swisher]

I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works.

[Rebecca Nagle]

I don't know.

[Mangesha Tigadar]

Listen to Skyline Drive how to live Forever on the iHeartRadio app, Apple Podcasts or wherever you you get your podcasts.

[Kara Swisher]

So let's get to the tech infrastructure that supports the creation dissemination of deepfakes. Analysis from Media Outlet Indicator found that new toy apps often use infrastructure from tech companies. Vs explain the relationship between deep fake porn sites, tech companies and the social media platforms and to what extent do these sites and nudify apps rely on big tech infrastructure? This was an issue when I was interviewing people who were creating all manner of nonsense around the insurrection on January 6 and companies moved quickly to take some of it down, but they were clearly the way people got to these different sites.

[V S Subramanian]

That's a broad question, Kara, so let me try and break it down into pieces. The first is depending if you're generating deepfakes at scale. So think the kind of thing Lori was talking about with Mr. Deepfake. He's trying to generate thousands millions of deep fakes. And so that requires computational hardware, what are called graphical computing units or GPUs which today are pretty scarce and expensive. And that scarcity and price doesn't look like it's coming down very much anytime in the near future. Those resources are heavily supported by cloud infrastructure providers. Again, I'm not naming any specific companies, but there are a number of companies that offer cloud services. So part of the onus is placed on them. If somebody is trying to use their service to generate this kind of stuff at scale, if somebody's trying to generate just a handful of deep fakes, they're probably not going to figure it out. But if somebody is generating millions and millions of them, or thousands and thousands of them, they may have some idea of what's going on. But that's not easy either, because there are many challenges for some of these firms in figuring out what's a deepfake and what's not, what's consensual, what's not. And so, as a first issue, the bad guys are continuously evolving. So that means that the actors in question are changing, the technology used to generate this stuff is changing. And it turns out, having studied deep fake detectors quite a lot and created some on my own with my lab, what we know is that small changes can make a big difference. So to give you an example, if I'm drinking this cup of coffee and there's a deep fake of me with part of my face obscured as I'm drinking that cup of coffee, it's harder in many cases for many detectors to figure out that video clip of me saying something which I probably never said is not true. Likewise, the definition. Carrie, I'm not speaking from a legal perspective, but for a normal person, what constitutes an act that's sexual in nature can vary a lot. A sexualized act versus an actual act of some kind of sex, very different. Also, the relative positions in which a sexual act is carried out, where certain body parts may be obscured, but other things may be clear. All that causes machine learning algorithms to detect deepfakes to have a lot of problems. So I think, you know, those are some of the challenges from a technical perspective in figuring these things out.

[Kara Swisher]

One analyst has said that some deepfake sites now offer APIs to people creating non consensual images and video generators. Explain what that means.

[V S Subramanian]

So when you go to a website, you're typing stuff in a prompt or a query, or clicking on some buttons and you get a response. An API is an application program interface and it replaces you with a piece of computer code, and that piece of computer code is interacting in a different kind of language with the server behind the website you're going to. And so if somebody is using an API to generate deep fakes, that suggests that they are generating it at scale. That suggests that they have a piece of code that is shipping requests automatically through that piece of code to the server and that server is sending back whatever they requested. In this case some kind of sexualized image that suggests somebody who's trying to do it at scale, who is, you know, who probably has his or her own databases of people they want to pose in these situations that they were never in to begin with.

[Kara Swisher]

Indicators analysis also estimates that the nutifier economy may be worth up to $36 million a year. Laurie, in your reporting on Mr. Deepfakes, you noted the financial incentives for the community members posting in the forum. Tell us about the incentives and talk about the money making Laurie here. And for the platforms that host this content too, although for people don't realize. Google says it doesn't allow apps that contain sexual content. Apple says the company's app store prohibits overtly sexual content. Both companies disabled searches for new to find in their app stores. We'll get to the workarounds in a second, but talk a little bit about the incentives, the financial incentives.

[Lori Siegel]

It's interesting because when we were looking at this, that wasn't a site, that was a platform. It was a whole ecosystem that enabled this site to exist. And that included platform liability from a lot of these different companies. So for example, you know, the incentive was you can go on. Not only can you view, you know, thousands of videos of sexually explicit deepfakes of your favorite celebrity, you could make your own. And this is why we believed Mr. Deepfakes was so dangerous. He was dangerous because he was creating an army of Mr. Deepfakes because now there are all these creators on there which is kind of pay to play. And there's a freemium model and then there's this premium model where you can pay to have specialized deepfakes of public figures who didn't consent made. And so people were probably making more money than the guy behind Mr. Deepfakes off of this platform. It was creating a whole ecosystem. But what else enabled that ecosystem? First I would go from a regulatory standpoint, the laws just hadn't caught up. And we saw this with non consensual pornography. Oftentimes tech moves quickly when there aren't the correct guardrails. Women and young folks are the first ones to feel that pain. But also there were payment providers that were accepting payments on this platform. And so it was all of these things and we talk about these small steps that actually helped enable this ecosystem to exist. And one of the coolest things we saw, because I think so many folks would say, well, it's a game of whack a mole. You go after this guy, and then there are all these other ones that are going to pop up. But actually what ended up happening is we did go after this guy. We found out who he was. And I think maybe most alarming was he wasn't a technical guy. This is a guy who was, I would say, in his 30s, worked at a pharmacist at a hospital helping people, was newly married, had a young child, and he had been able to create a website that enabled a whole new wave of deepfake abuse because he had an interest in technology and deepfakes. And he saw an opening because when Reddit banned sexually explicit deepfakes, we were able to track where his username was. Talking about seeing an opportunity that could be profitable and getting other people to build on this. And so all of these things together created, I would say, an ecosystem that allowed it and all of these things. Also, once people started chipping away at this, once the laws started slowly but surely catching up, once Google deranked the platform, there were all sorts of small steps that actually created a friction that made it harder to exist.

[Kara Swisher]

So let's shift to the current laws and regulations surrounding deepfake porn, including the Take It down act, the law that makes it a federal crime to publish non consensual, explicit deepfakes. The federal government began enforcing it in May. Carrie, in your pursuit of tech platform accountability, you've come up against section 230, as have many. The law that protects platforms from being held legally liable for content that users post. You mentioned product liability law as a way to stop deepfake porn and get justice for victims. So talk about using product liability law.

[Carrie Goldberg]

Section 230 was intended to basically just immunize platforms for content that, that third parties created and posted. But when it comes to deepfakes, especially like in the situation of Grok, we're saying that Grok was the one who created this material and also published it. And therefore section 230 shouldn't apply at all. But even if it does, by suing them under product liability, we're suing them not for publishing the content, but. But for basically harming people through its defective design, defective manufacturing. These were all foreseeable uses. There's no reasonably safe use for a nudifying product. And so that's just like a second way to overcome section 230. Now, Xai, they haven't fully exposed what their defenses are going to be in our case, but when they were opposing the temporary restraining order, they were claiming that. That they wanted. I mean, so far this would be the only AI product that is saying that Section 230 should apply to it and that they're not responsible for what people type in as prompts. But even more alarming is that they're claiming that they should have the First Amendment protections and that GROK should basically be protected under free speech, which is a pretty alarming idea. You know, that AI and a chatbot would have free speech rights. They don't. You know, so far we're not at a point where we're giving constitutional rights to a chatbot, but again, the arguments haven't yet hit the stage where they've been fully briefed yet.

[Kara Swisher]

So Vs talk about workarounds, because a lot of these companies are very clever in terms of workarounds. Porn people have always been very clever. How easy is it to do workarounds here from a technological point of view?

[V S Subramanian]

You know, I was thinking about this when Laurie was speaking earlier. Kara, you know, she talked about a website where somebody goes in and says, I want to create a deep fake of such and such person doing such and such thing. And so that's a classic example of a place where that website is using the APIs that you asked about earlier to access a cloud provider, where there's code running to generate the notification of the sort desired by whoever asked the query. But now think about it this way. Let's say the platform has the mechanism. Let's say they have a perfect mechanism that doesn't exist today. But suppose they did to detect a deep fake video of some form of sexual activity. Then the workarounds for the porn guys, there are some simple ones. So one of the classic methods to generate deep fakes, regardless of whether it's for good or bad, is something called stable diffusion. So in stable diffusion, what you do is you, let's say, throw in an image. I'll use an image rather than a video as my example. But you throw in an image of somebody and that image gets converted to some very coarse representation. You know, think of this as something that's a very skeletal version of that image. And then what ends up happening is that skeletal version goes through the nudification process, but at the skeletal level. So that skeletalized nudified image doesn't look anything like what a real nude might look like. And so you can think of it as some kind of computationally weird representation of a person who's about to be sexually abused via a deep fake. And then they could do all this on platform, on a cloud. And then without ever passing the source image or the final image, they bring back this coarse representation of the nudified image and perform the last step back on their own servers, where it generates the final nude. So that's an example of one way in which somebody with just limited computer science knowledge who's familiar with the code, who has access to all this code, can generate something while avoiding the guardrails of some of the platforms, even if those guardrails are close to perfect.

[Kara Swisher]

We'll be back in a minute.

[iHeart Podcast Announcer]

Run a business and not thinking about podcasting, Think again. More Americans listen to podcasts than ad supported streaming music from Spotify and Pandora. And as the number one podcaster, iHeart's twice as large as the next two combined. So whatever your customers listen to, they'll hear your message. Plus, only iHeart can extend your message to audiences across broadcast radio. Think podcasting can help your business? Think iHeart streaming radio and podcasting. Let us show [e-mail] that's iheartadvertising.com

[Unnamed Historian/Commentator]

the declaration, which is full of these beautifully rendered, you know, sentences and paragraphs about enlightenment ideals, does also have this darker history to it.

[Rebecca Nagle]

Why is it important for the darker part of the Declaration of Independence and the American Revolution? Why is it important that Americans know about it?

[Unnamed Historian/Commentator]

Well, if we don't understand the full context in which our nation was founded, we won't understand the full context in which our nation now finds itself itself.

[Rebecca Nagle]

I'm Rebecca Nagel Gohin Tawn J la citizen of Cherokee Nation.

[Lori Siegel]

Are you guys big Chiefs fans?

[V S Subramanian]

Hell, yeah.

[Rebecca Nagle]

This is First America, the true story of how the United States came to be and how we got to this present moment. Listen to First America on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Ben Nadifray]

Thirty years ago, an island off Manhattan almost brought down New York City.

[Narrator (Revisionist History Teaser)]

Who will you trust? Your friends and neighbors? Who will you trust? The people five miles overseas.

[Ben Nadifray]

This is a story about neighbors turning on each other and what happens when a forgotten place decides it's had enough.

[V S Subramanian]

We reached for a feeling that exists, exists in this city. But we're not stopping, are we?

[Ben Nadifray]

I'm Ben Nadifra, and for my new series on revisionist history, I took a deep dive into Staten Island's attempt to secede from New York city. In the 1990s and what it all has to do with the politics of resentment that dominate America today. Listen to revisionist history, the Staten island problem on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Mangesha Tigadar]

I'm Mangesha together and I'm back with a new season of the podcast Skyline Drive. This time I'm diving into a rabbit hole of peptides, organoids, blood boys, blue zones and brain replacement to try to understand what this longevity obsession is all about and what it really means to live for forever, for all of us. I learned about some rad science.

[Joanne Chu]

I can make a brain for you and then we can test what drug is the best for your brain as opposed to his brain.

[Mangesha Tigadar]

Here are some hard truths.

[V S Subramanian]

I would expect Indians to age faster, but I did not expect it to be almost a four to five year

[Mangesha Tigadar]

acceleration and get myself into a world of trouble.

[Kara Swisher]

I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works.

[Rebecca Nagle]

I don't know.

[Mangesha Tigadar]

Listen to Skyline Drive, how to live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[Kara Swisher]

The Defiance act, which is aimed at allowing victims of non consensual deep fakes to sue for damages, passed in the Senate and is currently stalled in the House. Even though some states have criminalized deepfakes, enforcement remains low. Laurie, what extent are these laws effective when it comes to stopping or at least curbing the spread? It's happened before with credit card companies and certain businesses. You know, things tend to slow things down. But in this case, how important is it to have the take it down and the Defiance Act?

[Lori Siegel]

I'll answer, and I'm curious for Kerry's response too, but I think it's incredibly important. You know, the Take It down act, you know, force tech companies to pay attention and quicker. It gives them 48 hours to take down non consensual abuse. But oftentimes when it comes to victims, it's really difficult to find recourse for this, you know, and so the Defiance act would create civil penalties that would enable them to actually have the ability to go after the folks who have done this and have civil recourse, which I think makes a big difference. And when we started out on our search for Mr. Deepfakes, which was, let's just be honest, really a search for asking why is this allowed to happen? You know, there weren't a lot of state laws and I will say a lot of the state laws and a lot of the victims and survivors we spoke to, unfortunately, the onus in many ways becomes on these survivors to speak out about it and help change laws. And they did in the state level, which, you know, the state laws changing, all of them varying in scope. Civil criminal penalties actually were where a lot of these different victims initially had recourse. Which is also scary to watch as folks go after regulation at the state level because often times the federal government is just slow to move. I would say I hope that the Defiance act passes. It can get through. I think this is a bipartisan issue because it's really a child safety issue as well.

[Rebecca Nagle]

Right.

[Kara Swisher]

Which is how they're saying it. So, by the way, a Wired investigation found that GROK is still being used for non consensual explicit content. This is despite regulatory scrutiny and multiple lawsuits. And after Musk's X AI said it would add restrictions in response to an NBC investigation, the X Safety account posted, we strictly prohibit users from generating non consensual explicit deepfakes and from using our tools to undress real people. Explain how some of these legal dynamics apply to XAI in that regard, since you're in a lawsuit with them when it keeps showing up that they're doing the same thing you would accuse them of.

[Carrie Goldberg]

Well, I mean, that statement that they say that it's against their regulations and they're basically passing the blame to users. There's nothing in that statement that suggests that they are modifying their product so that deep fakes are no longer easy to create. And the problem with things like the Take It down act, it's great. You know, suddenly now there's a 48 hour window of time that tech platforms are supposed to remove illegal content, but it has no teeth because victims can't enforce it. Victims can't soothe the platform and say, hey, you know, it's now been 48 days and that content is still up. So the problem is with all regulations and all these options is that the victims cannot enforce them. And so when we're talking about who's the bad guy, VS has talked about the bad guys in these situations. In my view, the worst guys are the platforms. And so until we have regulations that put the victims in the driver's seats to not just, you know, get criminal justice against the individuals, but to actually get justice against the platforms, you know, I think the only recourse we have is to use our courts and make these companies pay.

[Kara Swisher]

So every episode we get a question from an outside expert. Here's yours. And you're all going to answer it, so let's listen to it first.

[Lori Siegel]

Hi, I'm Georgia Wells, a tech reporter for the Wall Street Journal. I just finished a deep dive into

[Carrie Goldberg]

the issue of teens using deepfake nudes

[Lori Siegel]

to harass each other.

[Carrie Goldberg]

And so the big question I would ask is that now that the government

[Lori Siegel]

has just started enforcing the Take It

[Carrie Goldberg]

down act, how might this change the harassment landscape for young people today?

[Rebecca Nagle]

Thank you.

[Kara Swisher]

Laura, you go first, then Kerry, then

[Lori Siegel]

Vs. Yeah, I do agree a little bit with Kerry on saying, you know, there's only so much you can actually kind of enforce when the onus is on the victim. I do think from a cultural standpoint it is very different than it was a couple years ago of federal law. Take It down act. Again, state law are at least sending this message that the stakes will be high and that young boys, young teenagers can get into real trouble for doing this and can get charged. I mean, I think that at least from a perspective of schools being able to speak to students, parents being able to speak to their children about this and having some structure will be helpful. Again though, I'm not sure the Take It down act is gonna be the thing that's gonna actually change this. I also agree with Kerry on platform liability and creating higher standards stakes for the tech companies and the whole ecosystem that enables this to happen.

[Kara Swisher]

Carrie.

[Carrie Goldberg]

So I think they'll get the memo that this is illegal. I mean, having criminal laws was really, really effective. When we were dealing with the scourge of image based sexual abuse over the last decade, we went from having three states to having 50 states with those laws. And my firm saw such an enormous and quick plummet of, of offenders creating images like that. So that's great, but we still don't have an apparatus that targets the platforms themselves and prevents them from basically monetizing this kind of humiliation. And until we have laws that victims can enforce themselves like it's not really harming the who, I consider the true offenders.

[Joanne Chu]

Right.

[Kara Swisher]

Which are the platforms themselves. All right, bs.

[V S Subramanian]

You know, I have very mixed feelings about criminalizing a 13 year old. All right. But at the same time we need to Protect perhaps other 13 year olds or 14 year olds or 10 year olds from the consequences that accrue to them because of some 13 year old putting out deep fake porn of that kid. So I think, you know, a first step has got to be something around education. A lot of our teachers in schools are simply not teaching their kids about deepfakes. And it's one thing if kids are using deepfakes or Generative AI to create reports for their class projects. I mean that brings up a whole bunch of other ethical issues. But, but it's another thing if they use deepfakes to explicitly cause damage to a kid and one of their classmates. So I think the ethics around this has got to be clearly articulated in class and by schools. And some of our schools may have done a good job of this, others may not. And you know that education and the consequences of those actions in a school context are clear. I think it'll be challenging, you know

[Lori Siegel]

Vs just to say, to respond if it's helpful just to respond to that. I think the frustration is so many times I agree. How do we expect 13 year old boys to know that this isn't okay when they're getting served up notification apps on social media? You know, I think it is. It's an and like two things can be true. And so there's a huge gap between education and being able to educate folks on this and what the platforms are allowing and what they say they're allowing and what coming through it creates a really difficult environment.

[V S Subramanian]

You know, I worry generally about the broad statement or the broad platforms moniker. I think there's a wide variety in what different platforms are doing. Some platforms are going to a great amount of trouble spending a tremendous amount of money to try and perform the best innovation possible so they can push put some of these problems to bed and others are doing much, much less. So there's like a huge range of what platforms are doing. Some doing relatively little, some making every effort to try and get there. Now I want to add that again, even if platforms were doing their very best, there's a famous saying due to a statistician at the University of Wisconsin, all models are wrong, some are useful. All right? And so whatever models are being used by platforms to find these deepfakes, they're never going to be perfect. And so the question is, are they making the effort? Some are making what I would consider a very good faith effort. Some.

[Kara Swisher]

Can you name who is doing a good job?

[V S Subramanian]

I'd prefer to stay out of that but I don't want to talk about specific platforms but. But yeah.

[Kara Swisher]

So let's end by looking at what individuals can do as deepfake technology continues to evolve. Carrie, now that the Take it down act forces tech companies to remove deepfake nudes, how do you request a takedown if you're targeted?

[Carrie Goldberg]

Well, part of what the Takedown act required was that there be a flow for people to easily request content removal. And so that kind of infrastructure. I really appreciated that because being incorporated into the laws. So ideally, you know, a victim within the, the platform itself can now report it. You know, whether it's through a URL or if it's like Instagram or Facebook, then the image itself would ideally have a method to request removal.

[Kara Swisher]

When digital forensics professor and deepfake expert Hani Farid was on the podcast in January, he expressed concerns that agent AI will supercharge the creation of deepfakes in the coming months. Vs. How do we make sure the new laws and policies don't block positive uses of these tools?

[V S Subramanian]

So I think again, we don't want to block the technology, we want the technology to evolve. Because again, as Laurie said much earlier in the podcast today, deepfakes do have the capacity to help do things like better understand disease. I'm not going to call this a deep fake, but the generative AI is being used to create new kinds of proteins that'll help cure diseases. Generative AI is being used to create materials that'll degrade and perform in ways that we would like that would protect the environment and more. So I think what we have to do is to look at intent. What is the intent of the individual who is creating a deep fake? And in the case of the 13 year old boy or 14 year old boy who's creating deep fakes of one of his classmates because he's pissed off about something, I think there, there's intent. Whereas if somebody, you know, there was this guy who created a deep fake of the Pope, Pope Francis in puffy robes and a very dapper looking version of him, you know, that I think is something that's laughable and I find that at least I find that somewhat entertaining. I doubt very much if the Pope was, you know, horribly offended. And so, you know, I think we want to allow artistic freedom and expression, scientific inquiry, but at the same time regulate so that it's the uses that are banned or that lead to the individual intent.

[Kara Swisher]

1. Yeah, Lori, in the end you tracked down Mr. Deepfakes what ultimately brought down the site and where is he now?

[Lori Siegel]

Great question. Where is he now? I would love to know. So eventually we did track him down with the help of, you know, investigative journalists, the Internet and some of our cybersecurity friends who did some pro bono work for us. Cause they have kids and they viewed this as a real threat. And we were able to track him down. And you know, Karl, I'll never forget confronting him on his way into the office. And I do remember Carrie who I've known because I've covered non consensual pornography before. I called her before going to his parents home to make sure that we did everything correct, just in case, you know, we didn't wanna create any legal liabilities. But I will never forget him looking at me. He knew who I was with so much disdain. And I know this is a delusional part of me, but there was a part of me that thought maybe we would've tracked this guy down and we could show up and say, do you understand the real harm this site is causing? You're a father. And I remember he walked right into the hospital, he wouldn't say a word to me and the doors closed. And I remember thinking at that point, well, we're gonna have to go talk about this to everyone who will listen to us. And that has to change because we have to change culture as a part of this. This can't be where the story ends. He can't just take himself off social media, which he did, and continue running this site. And so a lot of things happened which I think were really exciting. I mean we, I think were one of the first to show up, up and threaten his anonymity, say, you cannot do this type of abuse and remain anonymous. So that was friction. That exposure was friction. A law started changing when in the UK they started talking about different types of laws that would make it harder for folks to access Mr. Deepfakes. They cut off access to folks in the UK that created friction. A very valuable server went down for them. The Take It down act passed. The CBC published their investigation as well. Mr. Deepfakes finally went offline for good. And I think it was such a great lesson in all of these players that actually impact how we build out a better future. And the headline is Friction. It just created more and more friction. Now what's happened to him, I think is actually is just as important of a headline, which is nothing really. I mean he lost his job at the pharmacy. But we've talked to many folks, law enforcement. I actually spoke to a couple different folks in Canada. From the law enforcement perspective, there's no open investigations into David Doe, that's his name. And there's thoughts that maybe he's left the country, but that's it. And I think that in and of itself tells its own story.

[Kara Swisher]

Absolutely. So the last question, Deep Faith Foreign obviously isn't going away. So what's the best possible outcome? Each of you, what is your most optimistic case for how it pans out in the next five Years. And how we get there there. Carrie, you go first and then Vs And Lori, you get the last word. Carrie.

[Carrie Goldberg]

Okay, well, it should probably come as no shock to you that my best case outcome is that the companies that unleashed these harms and profited significantly from them should have to pay and there should be disgorgement and compensation for all the victims and it should be financially penalizing for these companies.

[Kara Swisher]

So that would be your best. And is that a possible outcome? Do you feel like that's a possible outcome?

[Carrie Goldberg]

I mean, we're on that road with litigation. We'll see how it works. I also, you know, I do condemn the platforms like the App Store and Google Play, who for many, many years were offering and selling nudifying apps and they just kind of were ostriches with their head in the sand pretending like they didn't think these things were harmful. I'm really sick and tired of platforms not doing anything until something becomes actually criminal. You know, they deny that it's harmful until there's an actual criminal law.

[V S Subramanian]

V s. You know, that's what I would like in place, some kind of criminal law that says, well, look, here are the kinds of behaviors and even intended behaviors or intended outcomes that should be criminalized. Here are the penalties that you will face if you engage in these acts. And this needs to be something that's at least in the US Nationwide. But more importantly, many of these actors are going to be overseas. And so what we need is not just US Law, but law in various countries and some broad international agreements, perhaps enforced by an organization like Interpolation, which polices things like child trafficking already. They should be authorized to do and investigate things like this.

[Kara Swisher]

That's a very good point. That's exactly where it goes is abroad. What happens is immediately it goes abroad. Laurie, last word.

[Lori Siegel]

Look, I agree with what everyone here has said. So I will end with saying, you know, my gripe with Silicon Valley and I have a lot of gripes with Silicon Valley is they're sitting there talking about AGI and what happens when AI becomes. Become smarter than us. And it doesn't matter. It really doesn't matter. Because here's what's happening here on earth. Over here. In reality, we have children who are ending their lives because of deep fake sextortion, and people are sleeping on it. My hope is that I don't have to shout into the ether for years and say, this is real harm, this is real harm. That we can get better education, that we can change culture to understand that when new technology comes quickly without the correct guardrails. The folks who are impacted oftentimes are the ones that don't have the voice, and we can make some changes from there. Now, it doesn't take to Kerry's point. Oftentimes it takes criminal law, if I'm putting sitting in my seat. Oftentimes it takes public pressure and investigations. I'd love to get to a world where Silicon Valley maybe has some different folks in it. And this isn't the last thought as people are having dinner and talking about AGI after taking a cold plunge. That's it.

[Kara Swisher]

That's a very good point. But you're gonna wait a long time for that, just so you know.

[Lori Siegel]

But I can manifest it. I'm manifesting it right here.

[V S Subramanian]

Go ahead.

[Carrie Goldberg]

All right, sure.

[Rebecca Nagle]

Why not?

[Kara Swisher]

Why not? Anyway, I really appreciate it, all three of you. It's a really important talk and we're gonna keep coming back to it again and again and hopefully see something happen by raising awareness of it. For sure. Thank you so much, all three of you.

[V S Subramanian]

Thank you. Kara. Pleasure to be here.

[Carrie Goldberg]

Thank you so much.

[Lori Siegel]

Thanks.

[Kara Swisher]

We've reached out to Xai and X for comment. They did not respond.

[Ben Nadifray]

Respond.

[Kara Swisher]

Today's show was produced by Michelle Aloy, Katherine Milsop, Madeline LaPlante, Duby and Kalyn Lynch. Special thanks to Lissa Sop, Anika Robbins and Julia Sharp Levine. Our engineers are Fernando Arruda and Rick Kwan and our theme music is by Trackademics. Nishat Kurwa is Vox Media's executive producer of podcasts. Go wherever you listen to podcast, search for on with Kara Swisher and hit follow. Thanks for listening to on with Kara Swisher from Podium Media, New York Magazine, the Vox Media Podcast network, and us. We'll be back on Monday with more.

[Ben Nadifray]

Thirty years ago, an island off Manhattan almost brought down New York City.

[Narrator (Revisionist History Teaser)]

Who will you trust? Your friends and neighbors? Who you trust the people five miles overseas.

[Ben Nadifray]

This is a story about neighbors turning on each other and what happens when a forgotten place decides it's had enough.

[V S Subramanian]

But we're not stopping, are we?

[Ben Nadifray]

Listen to revisionist history, the Staten island problem on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. 150 years ago, they were hunting us down to kill us.

[Unnamed Historian/Commentator]

And now they're hunting down immigrants to deport them.

[Rebecca Nagle]

This is First America, the true story of how the United States came to be and how we got to this present moment. Listen to First America on the iHeartRadio app, Apple Podcasts, or wherever you get your Podcasts.

[Mangesha Tigadar]

I'm Mangesha Teegular, and I'm back with a new season of my podcast, Skyline Drive. This time I talk to scientists, biopunks, curmudgeons, blue zoners, super seniors, and Goa's top cryotherapy lab to try to understand this obsession with living forever and what it means for all of us. And I get into a bit of trouble along the way.

[Kara Swisher]

I'd say probably start bone smashing.

[Ben Nadifray]

That doesn't work.

[Kara Swisher]

To make it look more defined. They say it works.

[Rebecca Nagle]

I don't know.

[Mangesha Tigadar]

Listen to Skyline Drive. How to Live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

[iHeart Podcast Announcer]

I've been hearing for decades that the markets can solve climate change. Today, we have more incentives for market solutions than ever. And emissions are rising on this season of Drilled Carbon Cowboys, the story of three market solutions colliding in one multinational boondoggle.

[Lori Siegel]

Gotta give Bruce and the guys credit.

[V S Subramanian]

They're Republicans.

[Mangesha Tigadar]

They don't give a shit about any of this stuff.

[iHeart Podcast Announcer]

Listen on the iHeartRadio app, Apple Podcasts,

[Kara Swisher]

or wherever you get your podcasts.

[iHeart Podcast Announcer]

This is an iHeart podcast.

[Kara Swisher]

Guaranteed Human.