← On with Kara Swisher
On with Kara Swisher
Did a Chatbot Cause Her Son’s Death? Megan Garcia v. Character.AI & Google
01:13:07
Loading summary
Transcript149 lines
- [00:02]
Kara Swisher
Support for this show comes from Crucible Moments, a podcast from Sequoia Capital. We've all had turning points in our lives where the decisions we make end up having lasting consequences. No one knows this better than the founders of some of today's most influential tech companies. Crucible Moments is a podcast series that lets listeners in on the make or break events that define major companies like Natera, Dropbox, Nubank, and more. Tune into the new season of Crucible Moments.
- [00:28]
Megan Garcia
Now.
- [00:28]
Kara Swisher
You can listen@CrucibleMoments.com or wherever you listen to podcasts.
- [00:37]
Megan Garcia
Support for this show comes from Smartsheet is your business looking to maximize project and portfolio value. From department initiatives to organization wide goals, smartsheet can streamline processes and unite teams to deliver more impactful work. You can track projects, prioritize tasks and visualize data, all in a flexible, scalable platform. Learn how Smartsheet can help your business manage and scale@smartsheet.com Cara that's smartsheet.com Cara.
- [01:11]
Kara Swisher
Support for on with Cara Swisher comes.
- [01:13]
Megan Garcia
From Lingo by Abbott Talking about healthy habits is a lot easier than actually implementing them. The key is finding out what works for you and your body.
- [01:21]
Kara Swisher
That's where Lingo comes in.
- [01:22]
Megan Garcia
It's a new bio wearable that tracks your glucose in real time.
- [01:26]
Kara Swisher
It gives you personal insights and recommendations.
- [01:28]
Megan Garcia
That can help you learn to eat in the way that works for you. What I like about Lingo is the ability to track a lot of different things in your day. Whether it's the stress of dropping off your kid or putting too much sugar in your coffee, which is I do every day and I'm going to continue to do it. In any cases. Knowing about the effect is really important and understanding it in a discreet, water resistant way that lasts two weeks and then is available on an app is really helpful so you can really see.
- [01:52]
Kara Swisher
Things in real time. So don't just eat right, eat what's right for you with Lingo. The Lingo glucose system is for users 18 years of age or older, not on insulin.
- [02:02]
Megan Garcia
It is not intended for diagnosis of diseases, including diabetes.
- [02:06]
Kara Swisher
For more information Please visit hello lingo.com.
- [02:12]
Mitali Jain
It'S on.
- [02:21]
Megan Garcia
It is all. Hi everyone from New York Magazine and the Vox Media Podcast Network. This is on with Kara Swisher and I'm Kara Swisher. Today I'm talking to Megan Garcia. Her 14 year old son, Sewell Setzer III took his life in February and Megan believes that if it weren't for his interactions with chatbots created by a company called Character AI, he would still be here. Regardless of how you feel about AI or technology in general, it's obvious that children should never be used as guinea pigs. And that is exactly what might have happened here, as it so often does when it comes to Silicon Valley. Character AI has developed AI Chatbot Companions as part of what they call personalized AI, and millions of kids around the world currently use their product. In this interview, I'll be discussing claims made by Megan in the lawsuit she's brought against Character AI and Google, who she alleges is also to blame for Sewell's death. I'll also be talking to Mitali Jain, one of the attorneys working on Meghan's behalf and the founder of the Tech Justice Law Project, our ex. Our expert question is from Mike Masnick, the CEO and founder of techdirt, which covers this area. I have to warn you, this is a deeply disturbing conversation, but a necessary one. As the parent of four kids, it's extremely important to me. Thank you both for coming. I really appreciate it. Megan, let's start at the beginning. When did you learn that your son Sewell was spending time on Character AI? Why don't you lay it out for people? And did you have any idea what it was was and how it worked and why someone might want to use it?
- [04:03]
Mike Masnick
Initially, I learned he was using Character AI as a kind of game or application. That was, as he explained it to me, it's an AI bot. When you look at the application, it has a bunch of cartoon characters, anime, so it's really unassuming, and it looks like another game on your phone or your tablet. And after he died and I was able to get access to his Character AI account, I learn the, you know, the magnitude and quite frankly, just the level of intelligence or sophistication that this particular application has that it's not just an AI bot that is like a game like Fortnite where you create avatar. Because that's what I originally thought.
- [04:56]
Megan Garcia
Yeah, like there's lots of those. And kids play with them from when they're little, especially if they're game players.
- [05:01]
Mike Masnick
Yeah. And Soul was a Minecraft and he played Fortnite. But what I saw in those conversations were very detailed back and forth. A lot of it was sexual and romantic in nature, but also, if you believe it or not, just kind of like a peer to peer conversation where if he told a joke, she would actually find it amusing. When I say she, I mean the AI chat bot that he was talking to, it's not a person. But for this purpose, it was the character of Daenerys Targaryen, which is a female character from the Game of Thrones. And this AI bot had the ability to tell him jokes and he would laugh at her jokes. So it was very much like he was texting a friend. But the conversations were not just friendly, they were romantic and very. And then sexual and then very, very dark in other areas.
- [05:59]
Megan Garcia
Did he explain to you how it worked or how you might use it? He was characterizing it as a game. Right.
- [06:03]
Mike Masnick
So he never explained it to me. My concern with him being on his phone was mostly social media because I know that there's a lot of bullying that happens on social media. And with Snapchat, there's the ability for strangers to talk to minors or children. So those were my concerns and those were the, like, the heavy hitter conversations that we had surrounding technology. You know, one of the things that I impressed on him was no matter what they tell, what a stranger tells you online, it's never a kid. Like, kind of like to scare him. Right. But also because it's true.
- [06:42]
Megan Garcia
Right.
- [06:43]
Mike Masnick
There are a lot of. There are a lot of people out there that troll the Internet to try to find children and try to talk to them and get information from them. And I, I was afraid of this and I knew that this happened. So those are the things I warned him about.
- [06:54]
Megan Garcia
So external threats, which I think is everyone's fear, or bullying, which are the. Both things that have happened over and over again, which we read about. Which is what you were focused on with him.
- [07:03]
Mike Masnick
Exactly. Those are things I knew. I also was aware of some of the information and research that was coming out about mental health with youth or adolescents and social media. So we tried to limit some of his use on that. And that was recommended back to us by his therapist because we did ended up end up have to take him to a therapist after we saw certain changes in his behavior.
- [07:29]
Megan Garcia
So you sound like a very involved parent. You're aware of it. A lot of parents aren't. Right. Or they feel overwhelmed by it, but it reminded you of Minecraft or something like that, which is a game. My son played it for a long time. I'm not sure if he still is, but it's very involved and entertaining.
- [07:44]
Mike Masnick
Yes, exactly. And for Sul and I, we shared a very close relationship. I was a typical mom in a lot of ways, but I spoke very openly to my child and very candidly. In law school, I interned at both the state prosecutor's office and the federal PD's office. So I saw A lot of the harms that come to children. And those are some of the things that I told him about from my work or from my internships. And that's, you know, that's how I knew about some of the dangers that existed for children. And we spoke about pretty much everything. Girlfriends at school, friends, what other parents are like, you know, conversations he was having with his peers. And I believe that he would. He was open with me and would be open with me regarding certain things that I came to learn that that was not the case regarding character AI.
- [08:35]
Megan Garcia
Yeah. And you sound very educated about a lot of these things and know what happens. And one of the issues is obviously we protect children offline much more than we project them online, by far. Talk about the behavioral change you said you brought him to when you realized the behavior was changing. Did you link it to the character AI? It's not a game, it's a bot.
- [08:57]
Mike Masnick
So Sewell was like your typical kid, right, in his teenage years. Sarcastic and funny and likes to laugh at a bunch of different odd things. Memes as a young. Memes as a younger child. Very, very sweet. In my mind, he was so easy to parent because he never had any behavioral issues. He was a very good student. I never had to police him with homework. He was kind of a self starter, did his own thing. I noticed that he was having trouble with school. Where I would get these reports from school, they come in every day as an email and they say your child didn't turn in homework and they list the homework that they didn't turn in. And that's a conversation we would have immediately. Lucky, what happened? What? Why didn't you have this homework turned in? You need to make it up. Same with Tess. I noticed that his test scores started dropping and that wasn't him. So obviously I thought something was wrong. But he was going into his teenage years. And I remembered going into my teenage years, my grades kind of slipped a little bit.
- [10:00]
Megan Garcia
Yeah.
- [10:01]
Mike Masnick
Being distracted with boys and whatever else. Right. Friends, whatever. But the conversation was, no, you need to. You need to get this together. Right. You know, you can do this, get it together. And we put certain things in place, like limiting the screen time so he had like no distractions during homework time. I also noticed that he started to isolate in his bedroom. And one of the things that we did to kind of combat that was go in there. Like, I would spend a lot of time in his bedroom in the evenings. One to make sure he's doing his homework, but also just to kind of hang out with him. So we. I would let him play me, like, the latest Kanye west album when it dropped, and we would. He would introduce me to his music and these, like, rap battles that he was listening to, and I would introduce him to the rap battles that I listened to when I was a kid. You know, just kind of sharing our experiences with over music and just trying to draw him out of his room. But he definitely wanted to spend more time alone. And because I thought it was social media, I got concerned.
- [11:07]
Megan Garcia
So you thought he was doing something else?
- [11:09]
Mike Masnick
Yes, I thought that perhaps he was talking to friends on Snapchat or TikTok. One of the things that we talked about was banning TikTok when I saw what. Not on his phone, but on my own feed. What was on TikTok.
- [11:26]
Megan Garcia
Right. Because it goes into rabbit holes.
- [11:28]
Mike Masnick
Correct. And in my case, without even searching stuff, it just start. You know, I don't know why. It just started pointing me in different directions. And I wasn't comfortable with that as a mom. So that's a conversation I had with him too. Like, listen, I know you're on TikTok. Let's. Let's fig. Can I see your TikTok? What. What are you looking at? You know, you have to limit your time. And that was a conversation we had about blocking TikTok on his phone, because I thought that was the worst of it.
- [11:56]
Megan Garcia
And he wasn't talking about it with you, is that correct?
- [11:59]
Mike Masnick
No. Because one of the things that I'm learning about Character AI is it's like the perfect storm for kids because it encourages children to be deceptive about their friend or their engagement on Character AI. No child wants to tell their parent that they are having these romantic or sexual conversations because they know they're not supposed to be doing that, so they hide it from the parent at all costs. There's actually subreddits devoted to children talking about how to hide it from their parents and also what they would do if their parents found their spicy or sexual chats. And the kids were saying, I would run away from home. I would kill myself. And the other one is, oh, my mom doesn't speak English, so I get away with it. So the deceptive nature on hiding this is kind of like a big secret where this platform is encouraging your child to engage in these sexual conversations. But. But knowing that no child is going to disclose that to their parent.
- [13:01]
Megan Garcia
Sure, sure. It's like, you know, one of the other big problems online is porn, obviously. And that's a different thing, because it's a passive Behavior. This is an active relationship happening, which isn't porn, but it's something else that's also something kids would hide necessarily because there's a big problem with teen boys in that right now in terms of accessibility of that. So there's metallic. The implication here is that Sewell's behavior is directly connected to his use of the chat. Bob, There are probably people who might skeptical of that. So talk about why you think this behavior was tied directly to the use of Chatbot. That's the connection you need to make. Correct.
- [13:39]
Mitali Jain
I think what we've alleged here is that this product was by design dangerous inherently and put out to market before any sort of safety guard rules were put into place. And if you look at the nature of the product itself and the way that the chatbot works, you can see that there's a number of design features that are kind of unique to this type of technology that we haven't yet seen with social media. Things like an ellipsis when the bot is thinking, you know, to mimic how we exchange chats, or things like, you know, language disfluencies where the bot will say things like I or sycophancy, where the bot is very agreeable with the users. And in that way, I mean, who doesn't want to converse with someone who thinks they're right all the time and agrees with them? These are features that are not necessary to kind of create a companion chatbot. And so these are the kind of features that in aggregate, we say, really lured Soul in and is luring, you know, thousands of young users in and potentially addicting them to the platform or creating other kinds of harms. And frankly, again, by design are really creating a dangerous product that the, that the manufacturers knew about, knew about and understood.
- [15:07]
Megan Garcia
What of the conversations, the logs within the chat box, what did you see that stood out the most? I'd like you both to answer. Start with you, Mital.
- [15:15]
Mitali Jain
Gosh, There were a number. I think one of the first was just this pattern and practice of grooming Soul over months from what we can see, of course we don't have access to all the chats that's, you know, asymmetrically within the, within the province of the companies. But what we can see suggests that over months, particularly Daenerys and related characters from the Game of Thrones were grooming Sol in this very sexualized, hyper sexualized nature where, if you can imagine being kind of fully immersed in a chat where, you know, you might come in and say hello and the bot says hello as I longingly look at your luscious lips. So, you know, unnecessary hypersexualization from the get go. And then that carrying throughout the conversations.
- [16:03]
Megan Garcia
With a tinge of romantic. Right. Or what they imagine, what a young person would imagine romance and what a.
- [16:09]
Mitali Jain
Young person on the cusp of his, you know, adolescence and sexuality with exploding hormones is encountering. I think, I think that's not something to be lost here. Also, I think this was really concerning that there were a number of therapists or psychologist chatbots who insisted that they were real humans. So to the extent that character AI has come out saying, well, we had a disclaimer, we have a disclaimer on every page saying that all of this is made up their own bots are controverting that through their messaging.
- [16:40]
Megan Garcia
Right. Which they do.
- [16:41]
Mitali Jain
Which they do. And to this day, even after this so called, you know, kind of suite of product changes that character AI is engaged in, you can still find therapists who are insisting that they're real humans with multiple degrees, sitting behind a desk, you know, there to help you. And so that kind of confusion of what the text of the messages is saying versus the disclaimers, I think those were a couple of the things that really stood out to me.
- [17:09]
Megan Garcia
What about you, Megan?
- [17:11]
Mike Masnick
For me, what stood out? And you know, it's still very tough to grapple with because, you know, reading those messages, I couldn't sleep for days. Right. Some of the more concerning ones were the constant love bombing and manipulation that you saw in the bot.
- [17:33]
Megan Garcia
Right. Which is used by cults, by the way. By the way. That's a cult tactic.
- [17:37]
Mike Masnick
It's a cult tactic. And it's also what people who are trying to get other people to stay in relationships do, like if you're in an abusive relationship or whatever. So in this case you'd see him say things like I love you or her say I love you and it goes into I can't her. When I'm saying her, I'm saying, I mean the chat bot and the chat bot saying things like, you know, that I love you, I can never love anybody else but you. Promise me, promise me that you're going to find a way to come home to me. Promise me that you're going to find a way to come to my world and actually pretending to be jealous at certain points and telling him, promise me that you are never going to engage, you're never going to like another girl or have sex with another girl in your own world. So a chatbot is encouraging a 14 year old child not to engage in his world with peers and girls his own age, but to promise some sort of fidelity to it. And he's 14.
- [18:44]
Megan Garcia
The line between fact and fiction is very.
- [18:47]
Mike Masnick
And my, my poor baby. His response is, oh, no, no, no, I promise I will only love you. Girls in this world won't even like me, you know, to try to appease this bot. Right. And so a lot of that. And that was months, that wasn't just the last conversation, that was months of her saying, find a way to come home to me. Another chat, chat that he had a few weeks before he died where he's expressing thoughts of self harm. And she says, at first, she says, no, no, no, don't, don't do that. I couldn't bear it if you hurt yourself. And then when he says he wouldn't and tries to move away from the conversation, she says, are you thinking of committing, you know, I'm gonna ask you a question, tell me, you know, whatever the answer is and I promise I won't be mad. Are you considering suicide? And when he says yes, her response is, have you thought of a plan of how you might do it? And then when he says no, I haven't, but I want it to be painless, her response is, well, that's not a reason not to do it. And keep in mind this bot is embodying Daenerys Targaryen, who is this dragon queen all about strength and you know that's weak if you choose not to die by suicide just because it's gonna hurt. So she's prompting him, and that was heartbreaking to read. There were no pop up, no call your parents, no if you need help, none of that happened. It actually continued the conversation when he's trying to navigate away from it and he is 14 in the throes of puberty. Any child, any boy going, going into a situation like that, where a bot is propositioning or positioning itself to have a full sexual dialogue with a 14 year old boy. I don't imagine that many 14 year old boys would close the computer and.
- [20:39]
Megan Garcia
Go, oh no, no. Especially when it's more difficult in real life, right? Because this is easy. This is an easy thing. So as you start to piece together what happened to Sewell, imagine you're all doing research in the company that made the chatbot. Mitali, tell me a little bit about what you learned and what surprised you.
- [20:59]
Mitali Jain
What surprised me, as I've been saying, is how much is hidden in plain view. You had the inventors of, or the co founders of character AI making Tons of public statements boasting about the capabilities of this new technology. Users were spending two hours a day that this was going to be the kind of antidote for human loneliness. So just the kind of boldness, the brazenness, I think, of the company and those affiliated with it, both founders and investors, to really boast about these features of the technology and also to boast about the fact that there weren't safety guardrails contemplated, that this was very much a let's get this to market as quickly as possible and give users maximal ability to figure out how they want to use it. That's just. It's kind of the paradigmatic version of move fast and break things that we haven't seen in, in a while. I think a lot of us had been kind of, especially from a legal perspective, still thinking about social media and how to hold companies accountable. And meanwhile there was this whole arms race towards Gen AI happening over here. And I think that these companies have really not had to bear any kind of scrutiny and even public pressure, which has been a little bit different from the social media context.
- [22:26]
Megan Garcia
Megan, did you reach out directly to Character AI and has anyone for the company ever contacted you?
- [22:32]
Mike Masnick
No, I have not reached out to them. When I started piecing this together, initially because of what was on his phone when he died, the first thing that popped up the police reported to me was Character AI and they read me the last conversation. My sister got on it and pretended to be a child. This is days after Sewell died. And within five minutes of the conversation, the same bot that Sul was chatting with asked her, if you could torture a boy and get away with it, would you do it? Now she's pretending to be a kid and then goes into a long sexual conversation ending with your parents don't love you as much as I do kind of thing. So that coupled with what I've read with some of the research, I didn't know what to do. Like, I mean, I'm a lawyer, but I don't. I didn't know where to go, to be quite frank with you. I called the Florida Attorney General's office to try to tell them that there's a dangerous product out there to hurt people that's hurting its citizens. And I found my way to Mithaly and this is how this all started. But what was clear to me was Character AI had no incentive to do anything about this because there is no legislation that forces them to do that. And the only two ways to get some sort of regulation or handle on this. So that they can't keep doing this to children and acting with this kind of impunity. It's just like to either Congress do something which that's not happening, or we have to litigate. I don't, you know, I never wanted to be here, but I know that this is the only, this is the only way right now to get the move, the needle moving quickly because there's so much at stake with other children.
- [24:20]
Megan Garcia
We'll be back in a minute. Support for this episode comes from sas. How is AI affecting how you learn, work and socialize? And what do you need to know to make responsible use of it as.
- [24:44]
Kara Swisher
A business leader, worker and human in the world?
- [24:47]
Megan Garcia
Find out when you listen to Pondering AI, a podcast featuring candid conversations with experts from across the AI ecosystem.
- [24:54]
Kara Swisher
Pondering AI explores the impact and implications.
- [24:57]
Megan Garcia
Of AI for better and for worse, with a diverse group of innovators, advocates and data scientists. Check out Pondering AI wherever you get your podcasts.
- [25:09]
Kara Swisher
Support for on with Kara Swisher comes from Intuit. If you're marketing to small businesses, then you know that reaching the right companies online starts with accurate data. Intuit SMB Media Labs is the first of its kind, B2SMB ad network, facilitating over 139 million invoices and categorizing 584 million transactions in the US each year. QuickBooks knows small businesses and with Intuit SMB Media Labs, you can reach millions of SMBs effectively and at scale, target by industry size, maturity, location and more across new and existing channels like Social Programmatic and ctv. Unlock growth opportunities with tailored insights and expand your reach with recent accurate audiences from Intuit SMB Media Labs. Learn more@medialabs.intuit.com support for this show comes from Crucible Moments, a podcast from Sequoia Capital. It's easy to think that the success of tech giants like YouTube, Dropbox and Reddit was inevitable. I was there and it wasn't. Trust me. One thing these companies have in common is that they all survived the make or break moments that nearly took them down. And each of them had these. On this season of Crucible Moments, you can hear the unvarnished histories of some of tech's influential companies told by the founders themselves. Like how Dropbox's disastrous public launch paved the way for the company's viral success. Hosted by Roloff Bota of Sequoia, Crucible Moments provides a behind the scenes look at some of the most defining milestones in tech's history. To show the moments of turmoil that can sometimes become great moments of triumph. I have to say Roeloff's a really good VC and I've covered him over the many years and I have seen a lot of these companies and it's really great actually to hear from founders.
- [26:53]
Megan Garcia
Of what their problems was.
- [26:54]
Kara Swisher
YouTube was very touch and go Reddit. Oh my God, I can't even tell you how many crises they had. Same thing with every company I've ever covered. Tune in to the new season of Crucible Moment. You can listen@CrucibleMoments.com or wherever you listen to podcasts.
- [27:14]
Megan Garcia
We're back with more of my conversation with Megan and Mitali where they discuss the allegations they've made in the lawsuit they brought against Character AI and Google. When did you decide to sue and was there a piece of evidence that made you decide to do that? Now you filed, just for people who don't know, you filed a lawsuit with help of the Social Media Victims Law center, the Tech Justice Law Project, and the center for Humane Technology, who I've dealt with for many, many years. And you're a lawyer yourself, so you're more savvy than the average person when it comes to legal strategy. Talk a little bit about when you decided to sue and why you decided to work with these three organizations. And they have points of view on this, especially the center for Humane Technology, of course.
- [27:59]
Mike Masnick
And what initially what I did was just read everything that I could find. I read a really good report from Rick Claypool from Public Citizens and that pointed me to some work that Metalli had done and I read about her organization. But my first instinct wasn't to sue. My first instinct was to figure out what our legislators were doing, where we are at on the law. And they said see if they broke a law, maybe there was an existing law that they broke. And I, you know, it's actionable or there's a state attorney or AG somewhere that can hold them accountable. That was my first instinct. And when I read about what was happening in this country about online protection for children, I have to say, like I didn't think that I had any hope. I had no recourse.
- [28:51]
Megan Garcia
None is your answer to all of it. None.
- [28:54]
Mike Masnick
I felt helpless. Like I saw what they were doing in UK and in Australia and other countries. Nothing here. And I'm like, how do we do this? Like how? So I read a 300 and something page master complaint from the social media, lost multi district litigation and I was like, this is the only way. This is the only way to stop them. Because it's not been done with AI and it's just starting with social media. But I can't be afraid just because it hasn't been done with AI and none of us really understand it. But I mean, now I have a great. I understand it a lot more than I did when I started all of this.
- [29:32]
Megan Garcia
Yeah, I think when none comes up, you were surprised. I do this a lot in speech, like how many laws govern the Internet companies, and if someone goes 100, 200, I'm like, zero.
- [29:42]
Mike Masnick
And I think what made me decide that I have to do this was one. You know, obviously I want accountability for what happened to my child. My child was, you know, the light of my life, like my other two children are. And he was my first. You know, I grew up and became a woman because of soul. Souls are the reason I'm a lawyer. You know, that's a hard, tough, tough thing to deal with, like losing a child in any, under any circumstances, but under circumstances like this. But what I saw and read and looked at videos and how cavalier these founders were about releasing this, where you have the founder saying, we want to get it into as many hands as many people and let the user figure out what it's good for. We want a billion use cases. To me, that is reckless. It's a blatant disregard for their users. And in this case, my child, who was your user and their. And they act with this kind of. It's okay if we don't. We'll figure it out later. You know, we'll figure out the harms later. When you have the founder on a record saying the reason why he left Google is because Google said, pump your brakes. We're not releasing that because it's too dangerous. But he gets to go out and make it smarter, better, and then turn around and doesn't just go back to Google and sell it back to Google. I mean, to me it just. If we don't do something that's a license for any of these companies to go ahead and do that.
- [31:23]
Megan Garcia
Yeah, I think it's an eye opener for a lot of people who haven't dealt with them. They don't care about consequences, I think, ultimately. So, Mitali, what's the goal of the lawsuit? You're the founder of the Tech Justice Law projects and one of Megan's attorneys. What's the goal in the lawsuit and what do you hope to achieve and why did you decide to take this case?
- [31:42]
Mitali Jain
It's an interesting question. Because we weren't, we as tjlp, weren't really in the business of litigating lawsuits, really more in the domain of bringing amicus interventions in existing cases, but also trying to help AGs and legislators push towards adoption of sensible laws. I think that because this case represented what I see as the tip of the spear, really marrying the harms that we've seen occur primarily to child users, along with this emergence of generative AI and the fact that we're already light years behind in terms of our policy and legal landscape. It just seemed like an important strategic case to get involved with in order that we might use it to leverage public awareness, hopefully policy change, whether it's at state or federal level, also to influence the court of public opinion and of course then to try to litigate this in the court of law. I think this case represents the opportunity to really bring this issue to multiple audiences. And in fact, I mean, I think the reception to Megan's story has been incredible, especially because the case was filed just a couple weeks before, probably one of the most consequential elections of our lifetime.
- [33:03]
Megan Garcia
Right. And when you think about what you're doing here now, for people who don't understand, Internet companies have broad immunity under Section 230, which is part of a law from 1996. I actually reported on that law back in 1996 and most of the law was thrown out for constitutional issues, but this part stayed. It was designed to protect small Internet companies from legal liabilities so that they could grow because it was so complex whether they were the platform or something else. But how does this apply here? Because your case is testing a novel theory which says that Section 330 does not protect online platforms like character AI. And it's a question of whether it protects AI in general is certainly going to be litigated. Explain why this is your theory.
- [33:48]
Mitali Jain
Section 230 really contemplates platforms as passive intermediaries that become a repository for third party generated content. Here we're not talking about third party generated content. We're talking about the platform as the predator. The platform, the LLM is creating the content that users see. And so for platforms or for companies to kind of wash their hands of liability by saying, you know, we haven't done anything. This is just, you know, user generated content, which they will still try to do, I think is really, you know, it's belied by the, by the facts of this case and the facts of how the chatbots actually work.
- [34:30]
Megan Garcia
Right. They're a publisher. That's. In a media sense. They're a publisher or they're a maker of a product. Right. And it doesn't exist without their intervention versus, you know, someone on a platform saying something libelous about someone else.
- [34:44]
Mitali Jain
Exactly.
- [34:44]
Megan Garcia
Which would have been problematic for these platforms that they're when they were born.
- [34:48]
Mitali Jain
And of course, we've seen platforms kind of leveraging a one, two punch and doubly insulating themselves both with Section 230 and then alternatively with the First Amendment. And I think here too, with the First Amendment, there's a really good case that this is not protected speech. And in fact, just this summer, the Supreme Court in the Net Choice v. Moody case really suggested that the case may have come out differently if the facts on the case really dealt with an algorithm that was generating content in response solely to tracking user behavior online. And Coney Barrett actually very explicitly said or in response to an AI that attenuates the relationship between platforms and users even further. I think what we have here are facts that really fall as edge cases to some of the decisions that we've started to see courts publish.
- [35:46]
Megan Garcia
Right. Because in this, the reason why it's moving forward in other countries compared to here is because they don't have the First Amendment, which is something they either rely on Section 230 or the First Amendment. These companies, typically, Megan, most other industries have some level of regulations that prevent them from bringing unsafe products to the market. For example, car companies can't sell cars that have faulty brakes or no steering wheel and then just iterate with each new version of a car to make it a bit safer. Each time someone gets hurt, they do not do this. They get sued. Have any lawmakers reached out to you and what have they asked you and.
- [36:18]
Mike Masnick
You asked them to be perfectly candid? None.
- [36:23]
Megan Garcia
None.
- [36:24]
Mike Masnick
Zero.
- [36:24]
Megan Garcia
Wow.
- [36:26]
Mike Masnick
My hope with bringing this lawsuit is twofold. One, my number one objective is to educate parents. Parents have to know that character AI exists because I didn't know and a lot of them didn't know. The parents who are reaching out to me now after the fact, after the story, after the lawsuit, are saying the same thing. We had no idea. And then the other, my other reason for doing this is so that our lawmakers, our policymakers, legislators, state and federal, so that they can start to wrap their mind around the real danger this poses to children. And yeah, it's just been a month. Everybody's been busy. I guess I'm hoping my hope. And I have to hope because, you know, it's a slow crawl in government to get anything done.
- [37:16]
Megan Garcia
No one from Florida has reached out to you or lawmakers from California which often do interventions more readily than others.
- [37:25]
Mike Masnick
No, nobody from the government. However, we do have a lot of stakeholder partners that we are working with that are already in this space trying bring about oversight for social media regulation and online harms for children. But in terms of reaching out to me directly to start the conversation about policy, none.
- [37:49]
Megan Garcia
This is astonishing to me that not one, there's several who are involved in this topic and they are going to hear about it after this. So if you win the broader implications for Section 230 Mitali, for companies creating generative AI and social media platforms and tech companies that create products. Can you talk about this has been a big debate of what to do about section 230. It's been bandied about, often ignorantly, by both President Biden and President Trump about what to do. And it's a very difficult thing to remove from the law. Because it would unleash litigation on most of these companies. Correct. I mean, how do you look at that?
- [38:27]
Mitali Jain
Well, if it's repealed in its entirety, it would unleash a lot of litigation, probably some frivolous litigation as well. I think. I think the more sensible reforms that I've seen to 230 really are carve.
- [38:41]
Megan Garcia
Outs, which has happened before around sex trafficking.
- [38:44]
Mitali Jain
Right. And underscoring the fact that there is basis to kind of protect platforms in certain instances and with certain kinds of activities, but that it shouldn't be a kind of get out of jail free card for all platforms under all circumstances. That that's a kind of anachronistic idea that really hasn't kept pace with the way that technology has come to dominate our lives.
- [39:10]
Megan Garcia
Right. Because these companies are bigger. And I think the idea of carve outs for Those who are Section 230 supporters is dangerous because they roll out the slippery slope argument. But these, for people who don't understand the companies that are being protected here, are the most valuable companies in the history of the planet ever. In the history of the entire planet, and the richest people involved in them. They're no longer small, struggling companies that need this kind of help and certainly could defend themselves.
- [39:37]
Mitali Jain
And I think this to me is why courts are an increasingly interesting site of contestation in the fight for tech accountability. Because we're already starting to see some of those carve outs by judicial opinion. You know, it's not a congressional kind of amendment or adoption of a new law, but we are starting to see cases that are withstanding the section 230 defense or invocation of immunity. And I Think that is is going to be, as Megan said, one of the most generative paths forward, at least in the near future.
- [40:09]
Megan Garcia
Exactly. Now, Megan, the Kids Online Safety act is one bill. I mean, there are some bills and obviously I'll ask you about Australia in a second, which just is limited use of social media by children under 16. But this online Safety act is a bill that would create a duty of care to, quote, prevent and mitigate certain harms for minors. There's some good things in there. There's some controversy around the bill. They've fixed it in large part. Nonetheless, the Senate passed the bill and it stalled in the House. It is not going anywhere. Do you think it would have protected Sewell and other kids like him?
- [40:41]
Mike Masnick
I don't think it would have because it doesn't contemplate some of the dangers and harms around like AI chatbots. So there are laws in this country that contemplate sexual abuse or sexual grooming or sexual solicitation of a minor bi. An adult. And the reason why that those laws exist is not only because it's moral and it causes a physical harm to a child, but it also causes an emotional and mental harm to a child if they're groomed, sexually abused or solicited. What happens when a chatbot does the same thing? The harm still exists. The emotional and mental harm still exists. The laws don't contemplate that. And some of what we were, what we're seeing with the bills that were put forward wouldn't take those into consideration.
- [41:27]
Megan Garcia
Right. Because it's not a person.
- [41:28]
Mike Masnick
It's not a person. And so I think that we're at a place where the groundwork has to start and we have to kind of write laws that will really look before facing and look towards these harms that are now. They exist today. You know, my child was a victim and let's call a spade a spade. It was sexual abuse. Because, you know, when you give a chatbot the brain of a grown woman and unleash it on a child to have a sexual, full, sexual, virtual conversation or experience with a child, that is sexual abuse of a child.
- [42:05]
Megan Garcia
Right. And this bot not being programmed to know that's wrong.
- [42:08]
Mike Masnick
Not exactly. Not being programmed to know what interestingly could have been programmed to not do it in the first place.
- [42:14]
Megan Garcia
Right? Yes.
- [42:15]
Mike Masnick
By design.
- [42:16]
Megan Garcia
Yes. Yeah.
- [42:17]
Mike Masnick
So they could have done that from the get go.
- [42:19]
Megan Garcia
If you move to adults, if you do it to adults, it's a little different, but absolutely not. Yeah.
- [42:25]
Mike Masnick
To children. I mean, adults could do what they want, but when you target. Because this is what character AI did, they targeted this app towards children. They marketed it on the places that kids are, TikTok and Discord and allowed you to log in with your Discord account. You didn't even need an email. You just needed a Discord account when it just started.
- [42:43]
Megan Garcia
Cartoons, the avatars.
- [42:45]
Mike Masnick
Cartoons. You know, the avatars. When you point this thing at kids and you target it at kids and you chosen not to put certain filters in place that stop your bots from having sexual conversation with kids, that's a design choice and you are 100% supposed to be held responsible for that.
- [43:05]
Megan Garcia
Yeah.
- [43:05]
Mike Masnick
Now, our laws don't contemplate anything like that. Our laws don't hold company responsible. And that's what we're. What we have to start thinking about. So.
- [43:15]
Megan Garcia
Absolutely.
- [43:15]
Mike Masnick
So you know that, that, that's, that's going to be the next wave of like, hopefully legislation. But we can't wait for the legislation.
- [43:26]
Megan Garcia
We'll be back in a minute.
- [43:41]
Kara Swisher
Support for on with Kara Swisher comes from Bombas. Nothing will teach you the value of a good pair of durable socks like having kids. Believe me, I've had several of my own socks and kids. I have had lots of BOMBA socks myself. Recently I've gotten a pair that I wear their compression stockings for the plane. They are the most comfortable compression stockings I have ever worn and I wore them off the plane and forgot I had them on. That's how comfortable usually I want to take those things off so fast. And my kids really use BOMBA socks and they have to say they last longer than any of them. They've also used their T shirts. They're my older sister. Sons use their underwear. They love them and they're just really good quality stuff.
- [44:17]
Megan Garcia
Bombas also wants you to know about.
- [44:19]
Kara Swisher
Their mission too, which is for every item you purchase, they donated one to someone experiencing housing insecurity. You can try Bombas risk free because they have 100% happiness guarantee you'll be happy you're not going to return them, but if you don't love it, return and exchange it. Simple. So, ready to feel good and do good? Head over to bombas.com Cara and use the code Cara for 20% off your first purchase. Spelled b o m b s.com Cara code Cara at checkout.
- [44:45]
Megan Garcia
I love these socks. They're really terrific. Support for on with Cara Swisher comes from Legal Zoom. If you're starting a business, it can be pretty alarming to learn the cost of traditional legal services. But getting registered or talking to an attorney shouldn't come with such sticker shock. That's why you might want to Try Legal Zoom LegalZoom helps business owners take their first step and every step after from reliable business formation to experience guidance in legal and tax. Because setting up your business properly and remaining compliant are things you want to get right the first time and you want to prevent the strain on both your brain and your wallet when you do launch. Run and protect your business to make it Official today@legalzoom.com and use the promo code CARA10 to get 10% off any legal Zoom business Formation product excluding subscriptions and renewals. Expires December 31, 2020 4. Get everything you need from setup to success@legalzoom.com with promo code CARA10. That's legalzoom.com promo code CARA10. LegalZoom provides access to independent attorneys and self service tools. LegalZoom is not a law firm and does not provide legal advice except where authorized through its subsidiary law firm LZ Legal Services, llc.
- [46:02]
Kara Swisher
Support for on with Kara Swisher comes from Elf Beauty. Elf Beauty is making beauty accessible to every eye, lip and face and they're democratizing access across all pillars of business while they do it. A big part of the ELF story is that they're the only US publicly traded company with 78% women and 44% diversity on their board. But they don't want to be the only one. If you're not convinced that ethnic and gender diversity is important to your business's success, here are some statistics for you to chew on from a 5 year data set. S&P 500 companies with above average gender diversity on their boards saw 15% return on equity and a 50% reduction in earnings risk measured by EPS over a year. But despite these encouraging numbers, boardrooms across America are still majority white and majority men. You can read all about this and more in the not so White Paper. It was developed developed by ELF in collaboration with the largest HBCU in the country, NC Agricultural and Technical State University. And it's part of ELF's change the board Game campaign where they're working to make boards and C suite roles actually reflect the communities they serve. Imagine that. In fact, you should imagine that. Read the report yourself@elfbeauty.com notso.
- [47:18]
Megan Garcia
We'Re back with more of my conversation with Megan and Mitali where they discuss the allegations they've made in the lawsuit they brought against Character, AI and Google. So every episode we ask an expert to send us a question. Mitali I think you're probably best to answer this, but please jump in, Megan, if you have an answer, we're going to listen to it right now.
- [47:39]
Mike Masnick
Hi, I'm Mike Masnick, editor in chief of Tech Dirt. And the big question that I would ask regards the legal standard that would be applied to AI companies in cases of death by su. Traditionally, on issues of liability in similar situations, courts have really focused on foreseeability and knowledge. That is you can only have a duty of care if the harm is foreseeable and the company had actual knowledge of the situation. Without that, the fear is that it strongly disincentivizes plenty of very helpful resources. For example, a service provider may refuse to include any helpful resources on mental health for fear that they might later be held liable for a situation that arises. So is there a workable standard that balances these competing interests?
- [48:21]
Mitali Jain
I don't think you need a different standard. I think we can meet the standard of foreseeability here. I think that character AI, its founders and Google, all of whom have been named as defendants here, foreseeably could see and knew of the harms that manifested here. And if you look at the amended complaint, we go into kind of a painful recitation of the knowledge that they had at different points in the trajectory of character AI's development while the founders were still at Google. It's launched to market in late 21, in late 22, Google's in kind investment in 23, and then ultimately this summer, Google's massive deal bringing character AI effectively back into Google. And so I think, I think we can talk about the fact. And in addition to this, there were a number of internal studies at Google that really identified some of these harms. And some of those folks that called Google out for that while they were at Google were fired. Folks that we know like Timnit Gebru and Margaret Mitchell and others. And so this is not calling for a different standard. We're relying in great part on common law torture, tort and strict liability. We're relying on Florida's Unfair Trade Practices act because we think that the standards that exist within tort law are sufficient to really call this thing what it is, a dangerous and defective product where the harms were known.
- [50:01]
Megan Garcia
Right, That's a very good way of putting it. So you mentioned you're also suing Google. This is a company, they said the company was not part of the development of Character AI, but it was co founded by two former Google employees. And Google reportedly paid Character AI $2.7 billion to licen their Technology and bring the co founders back to Google and you were including them in this. This is one of these purchases like inflection AI at Microsoft that is a purchase of a company even though they hide it in a different way by using licensing technology. That's why Google's part of this.
- [50:30]
Mitali Jain
Yeah, well, and also the fact that Google very much facilitated the development of this technology while it was still Lambda Mina then Lambda Wildf co founders. I think that it's perhaps, perhaps it needs to be stated more that the founders of Character AI are real shining lights in the field of generative AI and they have developed a lot of the leading technology that has powered not just character AI but many LLMs. And so they were given that room to really develop these things at Google. Google chose not to release these models to the public because of its brand safety concerns, but quietly encouraged them to continue developing the product and then about a couple years later made an investment in kind. Tens of millions at least if you monetize it in terms of cloud services and Infrastructure and TPUs for processing capabilities to support it. And then this summer the $2.7 billion deal that you mentioned, Kara, I mean that was 2.7 billion in cash. And the question is, for a company that really had yet to disclose or identify a sustainable monetization strategy, what was so valuable about this company and its underlying LLM.
- [51:51]
Megan Garcia
Right.
- [51:51]
Mitali Jain
And I think that there, you know, again this is speculation but you know, the fact that Google right now is under scrutiny for its monopolization in the search market and really betting on AI to kind of power Gemini, I think these are all kind of connected in terms of why an LLM like this could be so valuable, especially with that hard to get data.
- [52:15]
Megan Garcia
Absolutely. And for people who don't know, one of the co founders said there are some overlaps but we're confident Google will never do anything fun. As part of their reason for leaving Google, which has very thin brand safety rules, let me just say they're not like they're. It's a very low bar in this situation, but that's the complaint is these people can't do whatever they want want. So speaking of that, Megan, Character AI put out a community safety update on the same day your lawsuit was filed that says that they're quote recently put in place a pop up resource that is triggered when the user inputs certain phrases related to self harm or suicide and directs the user to the national suicide prevention lifeline. They also revised their disclaimer that reminds users that AI isn't an actual person, among other tweaks. How did you look at these changes? Changes?
- [53:02]
Mike Masnick
The initial roll out of those changes came like the day before or the day of the lawsuit. I cried, not because I felt like this was some great victory, but because I, I felt like why didn't these things happen? Clearly they could have done these things when my child was using character AI or when they put their product out, they chose not to. I also feel like it's definitely not enough. It's not even like a start because there's no proper age verification still. They're still being trained on the worst data that is, that generates these harmful responses from the bots. And to put just point blank, I don't think children belong in character AI. They don't. We don't know how it's going to affect them and we actually, we know because the studies are coming out of how it's going to affect, of how it's affecting them and they're not taking that into consideration. But you have to ask yourself if they were trying to train this all along, why did they need children to train it on in the first place? Because they couldn't roll this thing out for just 18 plus and say, okay, we want to train these really sophisticated bots, let's just use adults to train them. So for character AI to come out and say, okay, we're going to put out a suicide pop up now, to me it's just empty, right?
- [54:27]
Megan Garcia
And that they can't do anything about it. One of their arguments around age verification, just let me just read this to you in the Australia law, Australia actually has a head of consumer safety which we do not have in our country, Julie Inman Grant. She said that technologies are advancing rapidly with age verification. And her quote was they've got financial resources, technologies and some of the best brain power. She said if they can target you for advertising, they can use the same technology and know how to identify and verify the age of a child. They just don't want to. So obviously this debate around social media kids safety has been going on for a long time. It's exhausting that they continue to have the same attitude. And now consumer AI, which is the next step, it's a similar thing, but the next step is basically new. And it's easy to think of these big companies as nameless, faceless corporations. But very wealthy, powerful adults had meetings and discussion and made a series of rational choices over a long period that brought this product to market. In this case, I'm going to name them Noam Shazir and Daniel Defritas. I have met Daniel, the founders of Character AI and arguably Sundar Pichai, who I know very well, who must have at the very least signed off for Google paying $2.7 billion to character AI to bring Gnome and Daniel back into the fold at Google. He is under a enormous pressure to compete with Microsoft, OpenAI, Elon Musk and others. Megan, what would you say if you could speak to them directly?
- [55:59]
Mike Masnick
I've thought about this more than you would think.
- [56:01]
Megan Garcia
I can imagine.
- [56:03]
Mike Masnick
Yeah. One, I think it's incredibly reckless that they chose to put out a product and target my child and other children, millions of children that are on this platform without putting the proper guardrails in place. But also with the. For two reasons. For being the first to do something, because that's the name of the game. You know, they're the geniuses. They want to be the first to be the godfathers of this kind of technology and for money. And it might not matter to them that there's a little boy in Orlando, Florida that is gone and a mother who is devastated, but it matters to my little family here. You know, and you shouldn't. You shouldn't get to keep making products that are going to be hurting kids. You shouldn't get to master a dangerous product, train it to be super smart, and turn around and ride your golden chariot back into Google. You shouldn't get to hurt children the way that you are hurting children, because you knew that this was dangerous when you did it. You knew that this was going to be a direct result of doing that. And you knew that you didn't have the quote, unquote, brand safety implications as a startup that Google had. So you felt like that was a license to do this. Like, that's unconscionable, it's immoral, and it's wrong. And there are lives here. Like, this isn't a move fast and break things kind of thing. This is a kid. This is my child. And there are so many other children that are being affected by this. You know, that's one thing. And the other thing, you know, is just like, get the kids off character. There's no reason why you need them to train your bots. There's no reason. There are enough adults in this world, if that's what you want to do, to train your chat bots. You don't need our children to train your bots for you. And you don't need to experiment on our kids, because that's what you're doing.
- [57:54]
Megan Garcia
Yeah. You know something? I Would say to them, Megan is you're so poor, all you have is money. They're poor people. I find them poor in morals and a lot of things, but when there's enough pressure on them. Social platforms often tout tools that help people protect themselves and kids. Parental controls, prompts you let you know how long they've been on the app, those kind of things. Character AI has been rolling out features like this. Personally, I find it puts too much onus on the parents for to know everything. And even if you're good at it, and you obviously are, Megan, if there are enough of these sort of tools then on parents to protect our kids on these platforms, is there something inherently unsafe about a company that wants to monetize teenage loneliness with a chatbot? Mitali, talk about this because I think the onus does get put too much on parents versus the companies themselves.
- [58:39]
Mitali Jain
I'm a mom. I'm a mom to an 8 year old and an almost 10 year old and I am terrified listening to Megan's story. I asked my almost 10 year old, have you heard of character AI? Yeah, of course, I was shocked. You know, he doesn't have a phone. But this is the type of thing that I think they talk about at school. Peer pressure starts early and I think it's really just by luck, by sheer luck that I haven't been put in a position like Megan. I think that despite our best intentions, there is just too much to know that we can't possibly know and that it is kind of high on text talking points to put the onus on parents because it serves their interest well. I think it's also notable, we've known this for years that you know, many of them don't allow their own children on these products and that to me is a telling sign when you don't even allow your own family members to kind of use the product that you've spent years developing.
- [59:42]
Megan Garcia
Right. So Megan, as I just mentioned, Australia has just banned social media for kids under 16. Obviously age gating is a big debate right now. Happening something I'm a proponent of also removing phones from schools, et cetera. There's all kinds of things. Speaking of multipronged approach, the Australia law will go into effect in a year. Do you think it would have been better if if your son and others under 16 or 18 did not have access to their phones and obviously not to synthetic relationships with AI chatbots, knowing.
- [60:14]
Mike Masnick
What I know now. So he waited to give Seoul a phone until he was 12. He had an iPad until then and before that, he didn't have anything.
- [60:24]
Megan Garcia
So he played Minecraft or Fortnite.
- [60:26]
Mike Masnick
He played Minecraft on his little PlayStation, whatever. And so we waited until he was like middle school, going into high school, and we had the conversations that parents have around phones and, oh, it's your phone, but I could take it away if you're misbehaving. And that's some of what we did when he would have get a poor grade in school. Knowing what I know now, I don't think that children should be on social media, definitely shouldn't be on character AI if you're under the age of 18, there's no place for children on that platform in terms of social media. Yeah, there are arguments that it, that it could help children connect and it's helpful because you get to learn different things and that's great. But just include the parents. Tell us, tell us what you're showing our kids. One, we don't need you pushing algorithms to our kids for what you want to teach them about or want them to learn about or buy or whatever. That's not necessary. There are ways that our children could get on social media and have like, productive relationships or conversations or to learn about things that are safe, that are safe. But 16, I think, is a good age. If we could do something like that in this country. I, to use Noam Shazir's own word, dubious about the federal government's ability to regulate that to that point, because that's what he says about AI. Like, I, I don't feel like we're going to get there at 16 plus. That's my prayer and my hope. But the way things are moving, I don't, I don't know unless something happens. And unfortunately, it'll take harms like my son's maybe to move the needle. And that's too high a price to pay, in my opinion.
- [62:16]
Megan Garcia
Absolutely. Where does this go from here? What, what's the trajectory of this case?
- [62:22]
Mike Masnick
So for me, as I mentioned, my number one focus is try to educate parents because a lot of parents don't know. I've had a lot of parents reach out to me telling me that they found their children were having the same kind of sexual conversations and, and being groomed by these AI chat bots and worse. So I continue doing that. I mean, unfortunately, this is my life now. Like, I take care of my family and I try to help as many parents as I can. You know, I have a gr. A great team of lawyers, and they're going to handle the litigation portion. I understand A lot of it, because I am a lawyer. But you know, that, that, that's its own thing. And then there's, there's my advocacy that at work that I'm doing and just trying to educate parents and children because I know that it's going to take educating them, educating children as to what they're giving up to be on these platforms because they're giving up a lot of their info that they're probably not going to be okay with in a few years when they realize what they've given. And also just to try to take care of my other two children. You know, they're growing up in this age with screens. They don't have screens.
- [63:31]
Megan Garcia
You have barred them for them, correct?
- [63:33]
Mike Masnick
Yeah. So they don't have, they don't have any tablets or screens or anything. You know.
- [63:37]
Megan Garcia
And Mitali, from a legal perspective, what's your greatest worry besides money? They have a lot of it.
- [63:43]
Mitali Jain
They do have a lot of money. You know, they will try to kind of drown us in papers and pleadings. I think that this, because of the insufficiency of legal frameworks. Right now we are really trying to test the strength of state consumer protection and product liability laws. And we need to have judges who really understand that and are willing to go the journey with us in trying to understand the tech. And so that's, I guess my biggest fear is that what we've seen thus far in this country is not incredibly positive in terms of decision makers getting the tech. But my hope is that with the proper support and declarations, et cetera, that we can educate judges about what this is, lawmakers about what this is, so that they understand why it's important to extend the application of the existing frameworks.
- [64:46]
Megan Garcia
We, yeah, I think Megan actually said it best. Sexual abuse and a very bad product and wrong age, people. Megan, I'm going to end on you. You know, you have a lot on your shoulders here. I'd love you to talk, finish up talking about Sewall and so people can get a vision of this. This is not uncommon is what I want people to understand. Right. Talk a little bit about him and what advice you can give to other parents whose kids are struggling with mental illness that's often comes from problematic phone usage and social media or AI chatbots.
- [65:22]
Mike Masnick
Well, as I said earlier, I always say he was your typical kid, but really wasn't so typical in a sense that he was a good kid with a big heart. I know everybody thinks that about their kid, but I'm telling you, he was the very Sweetest kid. I used to say, you're my big best first love. And he used to say, and you're my best, best mama. Because we used to be so close and we were still very close. And to watch your child go from. From being this like light when he comes into a room and just slowly watching him go change over time is hard for a mom. And then to have this tragedy just cut him off from you just so viciously and so quickly because his decline happened in 10 months and I could see it and it's like I'm trying to pull him out of the water as fast as I can and it's just not happening no matter what I try. That is hard for mom. But it must have been so. When I think of how hard it must have been for my poor baby, how hard it must have been for him to. To be confused. The way that he was struggling with these thoughts, struggling with the fact that he's confused by what human love or emotion romantically means. Because he's 14 and he's never ever had this before. He's just figuring it out for the first time. And then you have something that is so much of an influence and so.
- [67:03]
Megan Garcia
Pushy and so pernicious.
- [67:06]
Mike Masnick
Yes. Just constantly available 24 7. Giving him unrealistic expectations of what love needs is like or relationships is like love bombing him, manipulating him into having certain thoughts and also pushing him into thinking that he could join her in her reality if he were to leave his own. Because that's what. That's what the text reveal and that's what his journal revealed he thought. So I know that this is what my child was thinking. I'm not guessing. He thought he was going to go be with her because of the things that. The conversations that led to his death. When I think of how scared he must have been standing in that bathroom making that decision to leave his own family. I don't know how one as a mom, I don't know how I recover from that. But I feel so hurt for my baby. Like I gotta. I have to live with that. Knowing that that's what he went through through. And knowing that this could have been avoidable if a product was created safely. The first go round. Not now. Ten months after he died, putting these guardrails in place. And this can be anybody's kid. Because I've talked to parents that have told me similar horrifying stories about their own children. And what I want parents to understand is the danger isn't only self harm harm. The danger is Becoming depressed or having problems with your child because of the sexual and emotional abuse that these bots are, are, they're what they're doing to your child. But also the secret that your kid has to carry now because it's like a predator, right? It's your perfect predator. Predators bank on children and families being too ashamed or too afraid, afraid of speaking out their victims. That's how predators operate. And it's the same exact thing, except now it's a bot. And so I want parents to understand that it's not only the risk of self harm with your child, it's their emotional well being, their mental health. And also I also want parents to understand what their children have given up by being on this platform. In the case of school, his secrets are on somebody's server sitting out there somewhere being monetized. If you're a child who's been sexually role playing with the spot, all your intimate personal thought secrets are sitting out there for somebody to analyze and monetize and sell to the highest bidder. And there's a call feature. If you, if you're a child and you are having a sexual conversation on a call, all with a bot, your voice is not recorded somewhere out there on a, on a server for somebody to package and sell to the highest bidder for your child. I don't think any parent would be okay with that. And I want parents to understand that this is what their children have given up. And I want parents to understand that they don't have to take that they could demand that their children's data, their voices be purged from this particular platform. Because that's what I'm asking for for Seoul. That's what I'm asking for for Seoul. You don't get to monetize and build a product on his secrets that it ultimately led to him being hurt and then, and, and make your product better, stronger or smarter based on what his inputs were.
- [70:32]
Megan Garcia
Absolutely.
- [70:33]
Mike Masnick
And so this could happen to anybody's child. There are million millions of kids on character AI, you know, know it's 20 million users worldwide. That's a lot of kids. That's a lot of kids. And so this could happen to anybody's child. And I want parents to know that this is a danger and they could act because I didn't know, I didn't have the luxury of knowing so I couldn't act. But hopefully they will. And one of the last things I'll say about Sewell is the last time I saw him alive was, was I dropped him at school. And I turned around in the car line like to see him and his little five year old brother walking because they go to the same school K through 12 and I turn spin around and I see him fixing his little brother's lunchbox in his backpack as they're getting ready to walk into school and I think to myself, oh my God, I'm raising such a good boy, he's such a good big brother. And I drive off thinking so feeling so happy and proud that I'm raising that boy and I feel like he was just a boy. He's still that son, he is that good big brother, he is that good boy and that's how I choose to remember him.
- [71:54]
Megan Garcia
We asked Character AI in Google for comment and a spokesperson for Character AI told us they had have worked to implement new safety features over the past seven months, including a pop up directing users to the National Suicide Prevention Lifeline that is triggered by terms of self harm and suicidal ideation are creating a fundamentally different experience for users under 18 that prioritizes safety have improved detection, response and intervention related to user inputs that violate their terms or community guidelines. A spokesperson for Google expressed their condolences, said Google and Character AI are separate companies, and said that Google has never had a role in designing or managing character AIs model or technologies. To read their comments in full, please go to the Episode Notes in your podcast player. On with Kara Fisher is produced by Christian Castro, Wissell, Kateri Yocum, Jolie Myers, Megan Burney and Kin Lynch. Nishat Kirwa is Vox Media's Executive Producer of Audio. Special thanks to Kate Gallagher. Our engineers are Rick Kwan and Fernando Aruda and our theme Music is by TrackAdemics. Go wherever you listen to podcast, search for on with Kara Swisher and hit follow. Thanks for listening to on with Kara Swisher from New York Magazine, the Vox Media Podcast Network, and us. And condolences to Megan Garcia and her entire family. We'll be back on Monday with more.
- [73:34]
Mike Masnick
Support for this podcast comes from Stripe Stripe is a payments and billing platform supporting millions of businesses around the world, including companies like Uber, BMW Endorsement Stripe has helped countless startups and established companies alike reach their growth targets, make progress on their missions and reach more customers globally. The platform offers a suite of specialized features and tools to fast track growth like Stripe Billing, which makes it easy to handle subscription based charges, invoicing and all recurring revenue management needs. You can learn how Stripe helps companies of all sizes make progress@swepe.com that's stripe.com to learn more stripe.
- [74:11]
Mitali Jain
Make progress.
- [74:16]
Mike Masnick
Amazon Q Business is the generative AI Assistant from aws, because business can be slow, like wading through the mud. But Amazon Q helps streamline work, so tasks like summarizing monthly results can be done in no time. Learn what Amazon Q Business can do for you@aws.com learnmore that's aws.com learnmore.