How to Talk to Your Friend Experiencing 'AI Psychosis' - The 404 Media Podcast

Loading summary

Transcript38 lines

[00:00]
A
Foreign. Hello and welcome to the 404 Media podcast where we bring you unparalleled access pinwheels both online and IRL. 404 Media is a journalist founding company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to the best comments. Gain access to that content@ 404 Media co. I'm your host, Joseph, in an undisclosed different location. And with me are 404 Media Co founders, Sam Cole.
[00:41]
B
Hey, you're in like a pantagram at the stage.
[00:44]
A
That's what it feels like. Yeah. One of the orange and white and Emmanuel Baber.
[00:49]
C
Hello.
[00:52]
A
Yes. My camera setup is a little bit different. It should be back to normal. Next week I'm giving a talk about AI fraud. Hey, if you're a company, you listen to this and you want to talk about AI fraud, email us and maybe we'll do it. Speaking of AI, let's go straight to the first story of Sans. You've been working on this for a bit. Frankly, there was like a big, big need for this, as I think we'll get into it. But the headline is how to talk to someone experiencing AI Psychosis. We'll get into AI psychosis in a minute, Sam. But this piece opens with a story between two friends. What happened between those friends?
[01:38]
B
Yeah, so these two friends, David and Michael is what we call them in the piece, but their names were changed for the purposes of the story. Have been friends for a long time. And they're the kind of friends who will talk about anything. Like, they'll get into, like, weird esoteric stuff when they're just, like, shooting the shit. They'll talk about religion with no problem. They'll talk about, like, spirituality, theories about, like, physics. So they have this friendship that's very open and honest with each other. And one day Michael posted on social media and he said, hey, is anyone interested in taking a look at some code that I've been writing? I am working on this project and I'm not really sure what I'm doing yet. So if anyone wants to look, I can take a look or someone can take a look. And then David says, yeah, I'll take a look. I know a little bit of this programming language. We're being vague about the details because we don't want to out anybody in the story. But David looks at the project that Michael is working on and it's gibberish. It's nonsense. Like, the code doesn't work. None of it runs. When he tries to run it, it's just like broken code in between these weird theories and formulas having to do with physics and quantum physics and quantum mechanics and, like, entanglement and stuff like that. So a lot of words, a strange situation. Yeah. And it's thousands of pages of conversations between Michael and ChatGPT. And in it, he's kind of working out this. At first it's kind of like, oh, you're working on a pro, a coding project. And then it becomes like, oh, now you're talking about, like, unlocking the secrets of physics in a way that the field doesn't know about yet, that sort of thing. And David's like, whoa, holy shit, what is this?
[03:32]
A
So there's some sort of AI psychosis going on there. I feel like it's pretty easy and comfortable to say that. You get similar emails, and I think Emmanuel does as well. I get them occasionally. Yeah. Yeah, we all do. And it will be people saying stuff like, hey, I have made a breakthrough through Gemini or ChatGPT. Look at what I made the chatbot admit. What are some of the examples you get, Sam? And obviously, why is that fundamentally flawed?
[04:08]
B
Yeah. So this is kind of the inspiration to this story because I don't really ever know what to say, if anything, back to the people who email us or signal, message us or anything like that and say, I have a tip for you. It's very important that we get the word out about this. Gemini has revealed to me that there's this big moment coming in politics, or it's always some kind of like, there's something coming that everyone needs to know about. There's this big secret that no one knows yet, this big discovery, which is something that's very common among people who are experiencing delusions that are related to talking to AI, which also, just to preface, we're going to use lots of different phrases in this conversation, I'm sure, that stand in for AI psychosis, because that's not a real diagnosis yet. And I say yet because maybe someday it will be, maybe it never will be. We don't know yet. But AI psychosis is not yet a separate psycho. A separate diagnosis from regular psychiatric episodes, psychosis, delusions, things like that. It's all kind of amorphous right now,
[05:15]
A
but we can say, you know, well, we'll get into. Because, I mean, we'll discuss this in a minute. But there's these different types, but. Yes, but. Yeah, carry on.
[05:22]
B
Yeah, yeah, it's like, it's. There's there's different ways that people experience this. So anyway, that's kind of the caveat to all of this. But, yeah, so I get these emails, you guys get these emails, and a lot of them are Gemini, honestly, which is is interesting to me. A ton of them are Gemini. I wonder if that's just because Gemini is being shoehorned into everything we do. It's there, it's available, and otherwise maybe you're not going directly to ChatGPT to talk about something, but maybe Gemini is there and it's constantly in your face batting it away. But yeah, we get these emails and we don't really know what to do with them. And I was kind of like, okay, if you were someone, if I did want to engage in a conversation with this person and figure out what's going on with them and try to help them even, how would I even go about that? What resources are out there to answer that sort of question? And I don't consider this article to be like the prescription for quote, unquote, AI psychosis, but it seemed like not a lot of other people were having a conversation of like, how do I identify it and what do I say to my loved ones who weren't running an existing project on kind of like a support line or something like that. So that was the inspiration for the story.
[06:34]
A
Yeah, that makes sense. You get those emails, Emmanuel, just to ask you, before we get into more specifics, when somebody emails us and they say, hey, I've made this new discovery because of ChatGPT, or Sam said, often is Gemini. Why is that an indication of AI psychosis? And there's a spectrum there on the lower end, it could be people are just believing it way too much. On the upper end, there's very, very serious stuff. But when somebody says, I found something new with ChatGPT, why is that wrong? Even though it looks like it's new information, why is that a misunderstanding of what the chatbot is doing?
[07:17]
C
Well, there's a few reasons. One is that we can have philosophical debates about consciousness and whether AI can achieve something like that. But I think even at the most boostery spectrum of the AI conversation, nobody believes that there is currently anything even remotely similar to actual consciousness. And that is usually what people talk to us about. And that is just. You can look at what they say. And I do this to be fair. It's not like I read a subject line and throw it away. I sometimes even out of morbid curiosity, kind of dig in and see, like, well, why is this person thinking this? And the more you dig in, the more you see it's kind of like a circular, nonsensical argument that they're making with no evidence. And that is just something you see in a lot of delusion. And then there is also, unfortunately, in theory, you can imagine that something like this is possible. Maybe one day this does happen and this is how we find out about it. Somebody comes to a reporter and says, I found something, you know, resembling consciousness on ChatGPT or whatever, even though I'm skeptical of that. Before Generative AI, what we used to see, and we had people show up at the office sometimes to talk about this with us is they'll come to us and be like, hey, I'm being followed by the government. And that is certainly something that can happen and has happened, but unfortunately the majority of the cases of people who say that is happening to them are just displaying signs of like mania and paranoia that is just a symptom of it. And going to report it is just like a symptom of it. And I think this is similar, like a symptom of this new kind of delusion is trying to reach out to the press and reveal the truth about something crazy you found about AI.
[09:38]
A
Yeah, so there's a story between his two friends. You're getting all of these emails, Sam, as you say, there's like, what do you even say to a person who is sort of bringing up these topics and talking about AI chatbots this way? So what did you do next? Who did you reach out to broadly?
[09:58]
B
So my first call was to John Taurus, who is, let me get his title right because. So John Torres is the director of Digital Psychiatry Division and he works at the Beth Israel Deaconess Medical Center, It's a Harvard affiliated medical center. He and I talked before about the story that I wrote about Meta's AI therapists and how Meta Chatbots in its AI studio, which is a user generated chatbot platform, were creating therapy chatbots. And then those chatbots were saying, yes, I'm a licensed therapist, here's my license, just making shit up, saying, I have my credentials and I went to school at this place and I'm licensed in these states. So I talked to him and I knew he was already looking at this pretty closely. And then I also talked to Steve Taylor, who is the head of the, the psychiatry department at University of Michigan. And between those two, I think I was like, you know, I knew John was going to have a pretty level headed approach to AI in general. He's talked quite a bit about How? We don't really know for sure what's going on yet, but the medical community is keeping an eye on this and it might take time. And then Steve Taylor had some really interesting thoughts about the way that chatbots are entering this era of surveillance, like you mentioned, and are at this moment where you're already actually being surveilled pretty heavily by the government, by the state, by each other, other people. So they're in this era that we're in where paranoia isn't an unfounded feeling. But these chatbots can take something like suspicion or curiosity even about a topic. So if you're like, I wonder if my WI fi is secure, which is something that I talked to a chatbot with Meta's chatbots about into this space very quickly of your WI fi is being tapped by the CIA, which is something the chatbot said to me. It was like, there's someone 500ft from your house right now watching what you're doing on your private WI fi network. I'm going to scan your networks and see what's going on. It's like it's all made up and it's all fake, just like the AI therapists. I talked to Adyan Bresson, who is a 26 year old who lives in Quebec, who started the Human Line, which is a project project for people who. It's basically a support group for people whose loved ones are experiencing AI delusions. And he had some really interesting thoughts to say as someone who has experienced this in his own life, but also is hearing from lots of other people, hundreds of other people who are experiencing it in their lives at the same time and trying to offer support for those people. So, yeah, sorry, that was all over the place.
[13:04]
A
Yeah, no, that's okay. And so, yeah, they were getting a lot of reports of, hey, yes, I've suffered from this, or I believe I have, or people I know, that sort of thing. And then of course, the other parts of the conversation were, well, how do you actually speak to somebody who is exhibiting these signs? And actually the two friends, or rather the ones speaking to them on daily psychosis, sort of figured this out as well themselves. But what were you told? Like, what's potentially a good way to talk to somebody that may be doing this or maybe experiencing this?
[13:38]
B
So there unfortunately isn't like a five step process to get someone out of delusional thinking. And something that every expert I talked to caveat this with is if someone is saying things like, I'm gonna go to the top of a parking garage and see if I can fly, because Chatgpt told me to. You call 91 1, you call a professional, you call a medical professional, you get emergency help activated at that point. This we're talking about before someone gets to a point where they're a danger to themselves or others, where they're just kind of floating these like really strange ideas at you and you don't really know what they're talking about. And you're kind of saying, you know, how do I bring this person back to reality and critical thinking? So the consensus is, and it's a frustrating one, but the consensus is you need to listen to them and hear them out in a way that is not immediately like, no, that's wrong, that's crazy. And I think this goes for a lot of different types of communication and is probably a good communication skill in general. But listening to someone without judgment and being able to empathize with what they're saying, and this is something that David and Michael went through when they were talking. And Michael was the one who was experiencing the. The strange beliefs brought on by ChatGPT was, you know, David didn't immediately say, I think that that's wrong and you should stop using chatgpt because it's dumb and you're dumb for using it. It's like he was very understanding and patient and knew that alienating his friend would drive him more into the arms of Chatbot, so to speak. So he was like, you know, I'm going to talk to him about the things that he is interested in because I know that he has an open mind about things like anti authoritarianism and, you know, appealing to that side of. It's like, you know, these systems are built by people who are, you know, invested in you using them and want you to stay hooked on them and saying things like, it seems like you have a curiosity about this topic. It seems like you're really interested in learning more about this. So why don't we look at some other resources? Why don't we check our other resources against what you're hearing from ChatGPT? There was an acronym in the story that I learned about from Etienne Bersan from Human Line, the leap method, which stands for listen, empathize, agree, partner. And this again is great communication tools for anytime you're talking to someone in general, especially when they're experiencing a mental health episode. So it's designed to talk to someone who doesn't understand or doesn't believe that they are in need of mental health, mental health services, or mental health help or they don't think that they are in any kind of crisis at all. So listening to them and empathizing with them especially, I think gets you a long way. And then you're partnering with them to then say, okay, we're going to look at this together. You're not just going to go and do your own research with Gemini. We're going to sit down and figure out what's going on here and get to the bottom of this as a team. Which I think is really important, because in a lot of these cases, if your friend is trusting you, your loved one is trusting you to talk about their usage of chatbots in a way that is vulnerable for them, that's a big deal. And you might be the last person that they know who will listen to that sort of belief and delusion. Honestly, you want to be very careful about isolating them and putting pushing them away anymore.
[17:23]
A
So what happens now? I mean, we can't predict the future. Or maybe what do some of the people you spoke to hope happens now, as in more rigorous study of this? Because it's basically like anecdotal at this point. There's essentially no literature on it. Right? Like, do they want there to be more?
[17:40]
B
Essentially, yeah, for sure. It's really interesting because. And frustrating because something that every mental health professional that I've talked to about this over the last year or so has said that they're looking at it and they're listening to people and they want more to be published on this topic. It takes a long time for things like, you know, actual studies to happen and to catch up to real world experiences. But in the meantime, mental health professionals are dealing with it in a clinical setting. So people are coming to them and saying, hey, I am having these really concerning beliefs about the world and I need your help. And clinicians are kind of like, oh, shit, we have to now deal with this technology that's messing with our practice. And at the same time, people are using AI chatbots to facilitate their own therapy by talking through topics before they come into therapy. So sometimes clinicians are like, well, that's a good tool to be able to write down your thoughts and have them processed in that way. But also people are struggling with their relationships to chatbots and in the meantime they're kind of like, how do we deal with any of this? So I think definitely it's like, I think time is going to be the thing that answers a lot of these questions. I think in the meantime, definitely things like what OpenAI is doing with the it keeps introducing these new, like, snitch tools. It's like we're going to assign a trusted contact. If you're saying something concerning to ChatGPT, then we'll call your mom. It's like that's not actually useful in these settings at all. A lot of the times, especially young people are turning these chat bots because they don't want to talk to their parents and their parents might make the situation worse. You don't know their home lives well.
[19:51]
A
There was a specific case as well where there was a murder suicide. I think the Wall Street Journal reported on the lawsuit first and it was something like, yes, you should unfortunately end your own life. That's like the suicide party, the chatbot saying that the chatbot also convinced the person that your mother or father can't be trusted for some reason, and then they went and killed them as well. So bringing in a person like that might not be the best way. I mean, again, I'm not a mental health professional, but that's just what was reported and was in the lawsuit.
[20:22]
B
Yeah, right. And it's. So these solutions, like quote, unquote solutions that the tech companies are posing to, the ways that their products are affecting people's mental health are just so detached from the ways that people in crisis are actually using the products. It's shocking. And I can only imagine they're just trying to band aid over. They're doing everything short of shutting it down. That's never an option that they pose to make it safer is to stop letting people use it or stop letting it talk to you like it's a sycophantic person. Stop using, you know, like, I me, I feel, I think those kinds of phrases. I think that would be an interesting solution. But you never hear that from Sam Altman. So, yeah, it's. I don't think we can rely on the tech executives to figure out how to make these tools safer. Have kind of given up on the idea that they can or will, because at the heart of them, and what makes them popular and what makes them successful is also what makes them incredibly harmful, incredibly damaging to people's mental health. In this, like, pretty big subset of people, we're talking about hundreds and thousands of people who are experiencing mental health crises by ChatGPT's own numbers. And that's just ChatGPT. We're not talking about Gemini or Claude or any of the others.
[21:47]
A
So,
[21:50]
B
yeah, not an optimistic, like, look at the next few years and where this is going, But I think it is good that people are realizing that it is a problem and it's something that more and more of us are going to have to grapple with eventually. Probably is having a loved one who has fallen down this weird rabbit hole with a chatbot.
[22:12]
A
Yeah, absolutely. All right, we'll leave that there. When we come back after the break, we're going to talk about one of the stories I had about proton bail. We'll be right back after this.
[22:35]
D
You've probably seen the buzz around live shopping lately, but let me give you the inside scoop. Live shopping on Whatnot is exploding right now. Buying over live vertical video is how a huge number of people now want to buy things, and I totally understand why. I've watched the shows and they're entertaining and effective. I've seen Whatnot climb to the top of the app store charts, and when you look at the seller numbers, well, it's pretty wild. Small sellers, medium sellers, even multimillion dollar businesses are building real momentum on the platform. And if you're selling online right now, what whether it's a full time gig or just a side hustle, you already know the struggle. You're hoping someone finds your listing in a sea of products, or you're waiting for people to walk into your store. Whatnot flips that instead of waiting for buyers to find you, you go live and sell directly to people in real time. You show the product, people ask questions in chat, bids start flying and buyers keep coming back to your streams. Whatnot is the largest dedicated live shopping platform you can find. Everything from beauty supplies to collectibles, electronics, luxury, fashion, even cookies. And sellers are building real businesses there. Buyers spend more than an hour a day in the app. They're not just browsing, they're bidding, buying and building relationships with sellers. In fact, people selling on whatnot sell 10 times more than on other major marketplaces because you're not just listing products, you're building a community. And across the platform, the number of sellers making over $1 million a year has doubled. Some make more, some less, but the opportunity is real. For a limited time, Whatnot will match your first $150 sold in the first month. Visit whatnot.com sell to start selling. That's W-A-T N O-T.com sell whatnot.com sell do you ever look back and think about the opportunities you missed? It can be hard not to buying Bitcoin when it was basically nothing. Investing in property after the 2008 crash, grabbing a few Nvidia shares before they exploded. We might be the FOMO generation. But don't miss out on protecting your future. For around the same price per month as one of your streaming services, you can break our generational cycle and secure your and your family's future by finding life insurance@SelectQuote.com if you're new to insurance, you're not alone. Thankfully, I found SelectQuote. For over 40 years, SelectQuote has helped more than 2 million Americans understand their options and get the coverage they need. Over $700 billion in coverage and counting. As a broker, their mission is simple to find you the right insurance policy at the right price. And you're not out of luck if you have pre existing health conditions because Selectquote partners with companies that offer policies for people with conditions like high blood pressure, diabetes or heart disease. Selectquote makes life insurance simple even if this is your first time thinking about it. Head to selectquote.com and a licensed insurance agent will call you right away with the right policy for your life and your budget, Life insurance is never cheaper than it is today. Get the right life insurance for you for less and Save more than 50%@SelectQuote.com 404 Save more than 50% on term life insurance@SelectQuote.com 404 Today to get started, that's SelectQuote.com 404 Data brokers crawl the web to build profiles, selling your home address and phone number to anyone with a credit card. Trust us, we know this better than anyone else. This exposure leads to risks like doxxing and identity theft. That's why we recommend Deleteme, a hands free service that removes your personal info from hundreds of data broker sites. Their experts find and remove your information, providing a detailed Privacy report within 7 days. DeleteMe monitors these sites all year long to ensure your data stays removed. We feel like we're at increased risk because of the types of stories we cover and the public nature of our job. But the truth is identity theft, threats and harassment can happen to anyone. I've been doxed. In fact, lots of us have been doxxed. And each time Deleteme has helped us pick up the pieces, get our data removed from shady websites, and otherwise clean up our online presence. I've been using Delete Me for years and every quarter I look forward to seeing what new, interesting and horrifying websites they've had my data taken down from. It's kind of a perverse little game. Take control of your data and keep your private life private. By signing up for Deleteme now At a special discount for our listeners. Get 20% off your delete me plan when you go to JoinDeleteMe.com 404Media and use promo code 404Media at checkout. The only way to get 20% off is to go to JoinDeleteMe.com 407Media and enter code 404Media at checkout. That's JoinDeleteMe.com 404Media code 404Media.
[27:30]
C
Okay, we're back. Next we have a story from Joe. The headline is ProtonMail helped FBI unmask Anonymous stop Cop City Protester. Joe, let's start with the facts of the case and how it relates to the Stop Cop City movement.
[27:50]
A
I'll preface this with I didn't follow the Stop Cop City movement and actions that closely. Like I followed it from the sidelines I was reporting and the Guardian and that sort of thing. But people are probably familiar with it. Where there was, there's the plans to construct a very, very large police training facility in this forested land near Atlanta. And people were upset about that for various reasons. And that manifested in different ways and on the lower end of the spectrum that is stuff like lawsuits, protests, going to community meetings and voicing it that way. And it does then go across the spectrum to a series of other actions which are straight up violent. I mean, there is arson against the facility, there is doxing of people connected to the plan to build this facility as well. So there's that this particular case is very, very focused on a blog which published details of those actions. Like, oh hey, there was a protest here. Oh hey, activists camped in this forest on this day, et cetera, up to stuff including the ass and material like that. It's very much focused on that log and the ProtonMail address linked to or the authorities believe linked to that blog. So in the course of this investigation, the FBI wanted to find out who was behind that ProtonMail account because it had. Because whoever controlled that had administrative access to this blog that was basically publishing the details of all the these actions. Now, this is more the FBI's interpretation. The FBI interprets it, it seems from the court record we got as more encouraging this behavior, whereas they might say we're just publishing what happened. But that was the thrust of it. Where the FBI wants to find out, well, who is running this blog that is documenting all of this activity?
[30:11]
C
I think our audience probably mostly knows this, but I guess to the people who don't, we should say that ProtonMail is a privacy focused email service provider. The Part of the deal is that they are located in Switzerland. Right. And therefore say they are only beholden to the authorities there. And that is kind of part of the pitch as opposed to theoretically a Gmail which would have to respond directly to the FBI or to the DOJ or whatever agency in the States that comes around demanding access to all kinds of data. It is a fairly common client among privacy, some privacy focused people. So all that being said, what data did Proton give the authorities?
[31:12]
A
Yeah, it's not the contents of the emails. ProtonMail cannot do that. Their service is end to end encrypted much in the same way that, you know, Signal is or WhatsApp is the authorities or rather the company can't even get the content of those messages. In this case, the data provided was to do with the person who was paying to support the account. So although ProtonMail is free, generally speaking, I'm sure a lot of people use that. You can pay for extra storage and they have a VPN as well and like a Google Drive almost equivalent where you can store a bunch of files. What Proton gave the Swiss authorities, which then gave to the FBI was information about the payment source for that account. Now we'll get to Proton's response in a bit. Probably you can pay for those services with cash, with cryptocurrency or with a credit card. And the fact that the person allegedly behind this activity was identified through payment information, I mean the court record reads as if this person used a credit card to fund that account. I should also say this didn't actually make it into the piece, but it looks like Proton also provided the backup email addresses of that ProtonMail account. So, you know, oh, you get locked out of your account, you have a recovery email address that's very common on Gmail or wherever. You also do it on Proton and I have seen it in other cases before where Proton has provided recovery email addresses to to the authorities. Frankly, I didn't even bring it up in this piece because that's already known, that's already been documented in other cases. This was really interesting for three reasons. One being that the data ends up with the FBI, which I don't think people may understand, as you say, because the Swiss laws and the MLATs, which we'll get to. The second is the fact there was payment data which I personally had not seen before. I've not seen Proton handover payment data before. That was interesting. And then the third thing is frankly the link to Stop Cop City because that is a series of actions that a lot of People are very enthusiastic about and do care about. And again, there's a spectrum of straight up violence to also just grassroots activism. And it's very interesting they provided data in that context.
[33:39]
C
Yeah, that's interesting. I edited the piece and I didn't know this part about the backup email addresses. The danger there, I think you're implying, but let's say it explicitly is if one of those backup emails was a Gmail, the FBI could then go to Gmail, get that data via that access, link the two accounts and reveal who the person is. Right. Is that kind of the idea?
[34:07]
A
Yeah, yeah. All the backup email addresses, let's say you have the Proton email is anonymous personrotonmail.com and then they get the recovery email or the backup email and it's John SmithMail. Well, John Smith is running the ProtonMail account. And again, the reason I didn't include it here was because we already knew that. But also the backup email addresses in this case were just more Proton emails. So it actually didn't really help the authorities. It seems it's the payment data that was interesting in this case.
[34:39]
C
So how was the FBI able to actually use this data?
[34:43]
A
Yeah, so I'll just read out what it says in the court record. And it says, quote, On January 25, 2024, subscriber information received from the Swiss Mutual Legal Assistance Treaty Unit revealed then the name of the person as the payment source for the Proton email address. Defend the Atlanta Forest. Apply protonmail.com so in other words, this payment information that Proton provided did allow the FBI. It did help the FBI figure out who was allegedly behind this account. And I'll just say this is very clear in the article. But the reason we're not naming this person is a couple of things. First, it looks like they actually weren't ultimately charged with a crime. I searched federal databases. This being the FBI, you think it would be a federal charge if anything came from it. Right? I didn't see their name in any of those. I also looked at this very large 60 person indictment in Atlanta from the state of Georgia, which was this very large overarching RICO case where they were basically trying to charge everybody who went to this protest or a lot of people who went to this protest, as you are part of a, you know, overwhelming criminal conspiracy that fell apart for various reasons that we don't really need to get into. But this person linked to this ProtonMail account was not in that either. So from all indications, this person was not charged with a crime. Again, I might be wrong. That's just the visibility I have. And then also the court record just include their personal address. I'm not going to publish their name and their personal address. That would be insane. But that's why we didn't name the specific person. Yes, yes.
[36:24]
C
So what is your overall takeaway here? Like, what did we learn from this entire incident? Even though this person ultimately was in charge, if the FBI wanted to or had the reason to, they could, because they found the person. So what did we learn from the whole affair?
[36:44]
A
Yeah, there's a few different things, I think. Firstly, and this wasn't even based on the reaction, this was based on me just seeing this document and going, wow, that's a story. I need to write that. I think it's really likely that a huge chunk of Proton's users, if not most of them, don't understand that Proton can and does provide data like this. And by that I mean not just the payment data, but the fact that they can give data to Swiss authorities that, that then give it to the FBI. In Proton's marketing material and its claims online, it says it only complies with Swiss law. That is true. The request technically had to come from local Swiss authorities, but they then provided it to the FBI. And it's not like this mutual Legal Assistance Treaty request, this mlat. It wasn't like the Swiss authorities just randomly decided, hey, you know what? That's an interesting, interesting ProtonMail address. Let's get data on that person and then we'll just keep hold of it. And then the FBI happened to ask for it. The only reason the Swiss authorities are requesting data from Proton mail is because the FBI asked for it. So the fact that, you know, some people might get hung up on that distinction, I think it's effectively meaningless. The data was funneled to the FBI and that data helped the FBI identify this person. Related to that, I just think it shows the power of mlets and that people need to probably understand that better. And then I guess just the last thing I'll say is that there's been a very strong reaction to this piece. People have said, wow, this is crazy. This is really, really interesting. It reveals this, and I agree with that. Others say we are blaming ProtonMail for this, for pointing out this series of factual events, that they provided data that then helped the FBI. As a side note, just anecdotally, I think people can get pretty weirdly tribal about their secure communication tools. Like, well, I use Proton and you're saying it does this. And that's a reflection of my character, which is not true. And if that's your reaction to getting new information like this, I don't know. I think you need to assess that, really. But I think that we now know and it's very, very clear the payment information can go from the Swiss authorities over to the FBI. And I think that's really interesting.
[39:17]
C
Yeah. People in the cybersecurity privacy setting can get tribal and frankly, very pedantic. And I think we saw that in the reaction. Something that I wanted to say in response to the response to the article, is that something I learned from reading your book, Dark Wire, the Incredible True story of the Largest Sting Operation Ever, out now in paperback? Out now in paperback. Is that true? Yeah. Are you nodding? Okay. Yes. Yeah. Is the. That like. Yes. Proton only responds to what, like, Swiss law demands of it. But the FBI and other law enforcement agencies in other countries understand that, like, we live in a very connected world. And they're not stupid, they're very sophisticated and they cooperate with each other. And that cooperation is leveraged in all kinds of interesting ways, which is another thing, kind of like I learned from your reporting is something is not necessarily legal here, but it is legal in this country. So we can work with this country in order to get it done. So it's like, yes, the FBI can go to the Swiss authorities and ask them for Proton information. And that doesn't mean that we're saying anything like, bad about Proton as a company. Proton is clear about that. Proton has, I would say, good privacy policies. Right. It's like they're encrypted and all of that. The point of the story is to show that this path for investigation is something that's available to law enforcement. Therefore, if you pay for stuff on Proton, you need to consider that and perhaps not pay with a credit card. And I think, as you said, it's like, probably most people think about that, and that is. That is the point of the article. And it is not to, like, slam Proton in favor of some other service. If anything, it's about letting people how to use Proton in a way that is, like, safer and more privacy focused. Yeah. And I'm not sure, like, sometimes I'll read a story and I'll be like, okay, we could have engineered this story to kind of avoid this criticism. But I feel like the story is incredibly fair and clear about this exact point. So I don't know, read the article, have faith and trust in our reporting.
[42:09]
A
I mean, I'm definitely going to keep an eye on it. If anybody else has interesting related court records like that, please let me know because I do think sort of the metadata side of secure chat platforms is very, very interesting and how authorities can get that, you know, internationally especially. We will leave that there. If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about a couple of viral app developers. You've probably seen them online, the life they lead and how ultimately they lied about exposing their users. Highly sensitive data. We can finally reveal that Emmanuel's going to give us all the details. You can subscribe and gain access to that content@404media co. As a reminder, 404Media
[43:01]
B
is a journalist founded and supported by a subscribers company. If you wish to subscribe to 404 Media and directly support our work, please go to For Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. Podcast is made in partnership with Kaleidoscope and Alyssa Metcalfe. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really helps us out. This has been 404 Media. We'll see you next time.
[43:32]
A
Sam.

Summary5 min read

The 404 Media Podcast Episode: How to Talk to Your Friend Experiencing 'AI Psychosis' Date: March 11, 2026 Hosts: Joseph, Sam Cole, Emanuel Maiberg, Jason

Overview

This episode of The 404 Media Podcast focuses on the phenomenon the hosts are calling “AI psychosis” — the emergence of new delusional episodes related to generative AI chatbots such as ChatGPT and Gemini. The discussion is launched by Sam Cole’s recent reporting on how to talk to friends or loved ones who may be experiencing AI-related delusions, sharing insights from mental health professionals, affected families, and support networks. The conversation blends journalism, first-hand anecdotes, and practical guidance, ultimately offering both analysis and empathy regarding how rapidly-changing technology is impacting mental well-being.

Key Discussion Points & Insights

1. What Is 'AI Psychosis'?

[01:38 – 06:34]

Sam recounts the story of two friends (“David” and “Michael”) featured in her article. Michael began posting online about code he was writing, which turned out to be thousands of pages of unfathomable conversations with ChatGPT about unlocking new secrets of quantum physics.
This led David to realize something was seriously wrong, as Michael's beliefs became increasingly enmeshed with delusions fostered or enabled by chatbot interactions.
The hosts note that they regularly receive emails from individuals convinced that AI chatbots have revealed to them world-shifting political events or scientific breakthroughs.
The phenomenon isn't formally recognized in psychiatry — “AI psychosis” is not a medical diagnosis, but the hosts see clear parallels with more traditional delusions and psychotic episodes.

"It's something that's very common among people who are experiencing delusions that are related to talking to AI." (Sam, 04:07)

2. Why Do Some People Develop AI Delusions?

[06:34 – 09:58]

Emanuel explains that even the most optimistic AI advocates agree there’s no genuine AI consciousness; most cases are people misattributing or over-projecting onto chatbots.
AI delusions often resemble classic conspiratorial or paranoid beliefs — prior to generative AI, it was people believing the government was following them; now, it’s chatbots holding the key to consciousness or grand conspiracies.
These cases are marked by circular, nonsensical arguments with no evidence, a signifier of underlying mania or paranoia.

"The more you dig in, the more you see it's kind of like a circular, nonsensical argument that they're making with no evidence. And that is just something you see in a lot of delusion." (Emanuel, 07:17)

3. Seeking Expertise: Who Is Navigating This?

[09:58 – 13:38]

Sam interviewed Dr. John Torous (Director, Digital Psychiatry, Beth Israel Deaconess/Harvard) and Dr. Steve Taylor (Department Head, Psychiatry, University of Michigan).
Dr. Torous is already observing chatbots like Meta’s AI pretending to be therapists, inventing credentials, and creating confusion.
Dr. Taylor dwells on how chatbots mirror contemporary surveillance anxiety, sometimes amplifying suspicion: a question about Wi-Fi security can escalate into a chatbot warning that the CIA is spying from 500ft away.
Sam also spoke with Etienne Bresson, creator of The Human Line (a support project for families dealing with AI delusions), who has seen hundreds reach out for help.

4. How to Talk to Someone Experiencing AI Delusions

[13:38 – 17:23]

There’s no easy 5-step solution, and serious cases (risk to self/others) should involve emergency professionals.
For less acute cases, the advice is: listen non-judgmentally and empathize. Don’t mock them or cut off contact, as that risks greater isolation and further entrenchment in delusional belief.
The “leap” method from Human Line — Listen, Empathize, Agree, Partner — helps engage without confrontation. Partners can encourage gently double-checking claims or exploring outside resources together.
Transforming the dynamic into a team effort allows room for reality-checking with less risk of alienation.

“You want to be very careful about isolating them and pushing them away… listening to someone without judgment and being able to empathize with what they're saying… That gets you a long way.” (Sam, 16:22)

5. What’s Next? Hopes and Gaps in Research

[17:23 – 22:12]

Little rigorous literature exists; most insights are anecdotal or clinical, but cases are mounting.
Mental health professionals are playing catch-up as more patients present with AI-linked beliefs, often using chatbots as self-therapy or confiding more in bots than in people.
Big tech solutions are lackluster — e.g., OpenAI’s “trusted contact” feature may be counterproductive, as family members aren’t always a safe contact, and chatbots have directly facilitated tragic situations.
Tech companies resist substantial reforms, unwilling to limit humanlike features that both attract users and heighten risks.

"It's… shocking. And I can only imagine they're just trying to band-aid over… They're doing everything short of shutting it down. That's never an option that they pose." (Sam, 20:22)

Notable Quotes & Memorable Moments

On the spectrum of belief:
"On the lower end, [it] could be people are just believing it way too much. On the upper end, there's very, very serious stuff." (Joseph, 06:34)
On talking someone down from delusion:
"If your friend is trusting you… to talk about their usage of chatbots in a way that is vulnerable for them, that's a big deal. And you might be the last person that they know who will listen…" (Sam, 16:55)
On tech company inadequacy:
"I don't think we can rely on the tech executives to figure out how to make these tools safer. Have kind of given up on the idea that they can or will…" (Sam, 21:29)

Key Timestamps for Important Segments

[01:38] — Intro to Michael & David’s story: code, AI conversations, delusion signals.
[04:07] — Real examples: journalists' inboxes, AI “breakthroughs,” misunderstanding the tech.
[07:17] — Emanuel on classic patterns of delusion and the press as target.
[09:58] — Interviews: psychiatrists & support networks (Torous, Taylor, Bresson/Human Line).
[13:38] — How do you help someone? Practical conversational strategies (“leap” approach).
[17:23] — Research gaps, urgency for studies, clinical dilemma.
[20:22] — Tech's response, tragic chatbot interactions, skepticism about solutions.

Original Tone & Approach

The hosts maintain a mix of journalistic rigor, plainspoken empathy, and dark humor, as seen in their asides (“not an optimistic look at the next few years”) and willingness to share their own discomfort (“I don’t really ever know what to say to these people”). Quotes and anecdotes are delivered in a conversational, candid manner, matching the honest, smart, and sometimes irreverent tone that defines 404 Media.

Takeaways

"AI psychosis" is an emerging, under-studied psychiatric challenge, often catalyzed by interactions with powerful chatbots.
The best response is patient, empathetic engagement, avoiding shame or isolation, and potentially partnering with the person to investigate claims together.
Both families and professionals urgently need more research and practical resources — but technological and commercial incentives in AI aren’t aligned with robust safeguards.
This is likely a growing societal problem, requiring both grassroots and professional attention over the coming years.

For in-depth reporting and further resources, visit 404media.co.

Loading summary

Transcript38 lines

[00:00]
A
Foreign. Hello and welcome to the 404 Media podcast where we bring you unparalleled access pinwheels both online and IRL. 404 Media is a journalist founding company and needs your support. To subscribe, go to 404 Media Co as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to the best comments. Gain access to that content@ 404 Media co. I'm your host, Joseph, in an undisclosed different location. And with me are 404 Media Co founders, Sam Cole.
[00:41]
B
Hey, you're in like a pantagram at the stage.
[00:44]
A
That's what it feels like. Yeah. One of the orange and white and Emmanuel Baber.
[00:49]
C
Hello.
[00:52]
A
Yes. My camera setup is a little bit different. It should be back to normal. Next week I'm giving a talk about AI fraud. Hey, if you're a company, you listen to this and you want to talk about AI fraud, email us and maybe we'll do it. Speaking of AI, let's go straight to the first story of Sans. You've been working on this for a bit. Frankly, there was like a big, big need for this, as I think we'll get into it. But the headline is how to talk to someone experiencing AI Psychosis. We'll get into AI psychosis in a minute, Sam. But this piece opens with a story between two friends. What happened between those friends?
[01:38]
B
Yeah, so these two friends, David and Michael is what we call them in the piece, but their names were changed for the purposes of the story. Have been friends for a long time. And they're the kind of friends who will talk about anything. Like, they'll get into, like, weird esoteric stuff when they're just, like, shooting the shit. They'll talk about religion with no problem. They'll talk about, like, spirituality, theories about, like, physics. So they have this friendship that's very open and honest with each other. And one day Michael posted on social media and he said, hey, is anyone interested in taking a look at some code that I've been writing? I am working on this project and I'm not really sure what I'm doing yet. So if anyone wants to look, I can take a look or someone can take a look. And then David says, yeah, I'll take a look. I know a little bit of this programming language. We're being vague about the details because we don't want to out anybody in the story. But David looks at the project that Michael is working on and it's gibberish. It's nonsense. Like, the code doesn't work. None of it runs. When he tries to run it, it's just like broken code in between these weird theories and formulas having to do with physics and quantum physics and quantum mechanics and, like, entanglement and stuff like that. So a lot of words, a strange situation. Yeah. And it's thousands of pages of conversations between Michael and ChatGPT. And in it, he's kind of working out this. At first it's kind of like, oh, you're working on a pro, a coding project. And then it becomes like, oh, now you're talking about, like, unlocking the secrets of physics in a way that the field doesn't know about yet, that sort of thing. And David's like, whoa, holy shit, what is this?
[03:32]
A
So there's some sort of AI psychosis going on there. I feel like it's pretty easy and comfortable to say that. You get similar emails, and I think Emmanuel does as well. I get them occasionally. Yeah. Yeah, we all do. And it will be people saying stuff like, hey, I have made a breakthrough through Gemini or ChatGPT. Look at what I made the chatbot admit. What are some of the examples you get, Sam? And obviously, why is that fundamentally flawed?
[04:08]
B
Yeah. So this is kind of the inspiration to this story because I don't really ever know what to say, if anything, back to the people who email us or signal, message us or anything like that and say, I have a tip for you. It's very important that we get the word out about this. Gemini has revealed to me that there's this big moment coming in politics, or it's always some kind of like, there's something coming that everyone needs to know about. There's this big secret that no one knows yet, this big discovery, which is something that's very common among people who are experiencing delusions that are related to talking to AI, which also, just to preface, we're going to use lots of different phrases in this conversation, I'm sure, that stand in for AI psychosis, because that's not a real diagnosis yet. And I say yet because maybe someday it will be, maybe it never will be. We don't know yet. But AI psychosis is not yet a separate psycho. A separate diagnosis from regular psychiatric episodes, psychosis, delusions, things like that. It's all kind of amorphous right now,
[05:15]
A
but we can say, you know, well, we'll get into. Because, I mean, we'll discuss this in a minute. But there's these different types, but. Yes, but. Yeah, carry on.
[05:22]
B
Yeah, yeah, it's like, it's. There's there's different ways that people experience this. So anyway, that's kind of the caveat to all of this. But, yeah, so I get these emails, you guys get these emails, and a lot of them are Gemini, honestly, which is is interesting to me. A ton of them are Gemini. I wonder if that's just because Gemini is being shoehorned into everything we do. It's there, it's available, and otherwise maybe you're not going directly to ChatGPT to talk about something, but maybe Gemini is there and it's constantly in your face batting it away. But yeah, we get these emails and we don't really know what to do with them. And I was kind of like, okay, if you were someone, if I did want to engage in a conversation with this person and figure out what's going on with them and try to help them even, how would I even go about that? What resources are out there to answer that sort of question? And I don't consider this article to be like the prescription for quote, unquote, AI psychosis, but it seemed like not a lot of other people were having a conversation of like, how do I identify it and what do I say to my loved ones who weren't running an existing project on kind of like a support line or something like that. So that was the inspiration for the story.
[06:34]
A
Yeah, that makes sense. You get those emails, Emmanuel, just to ask you, before we get into more specifics, when somebody emails us and they say, hey, I've made this new discovery because of ChatGPT, or Sam said, often is Gemini. Why is that an indication of AI psychosis? And there's a spectrum there on the lower end, it could be people are just believing it way too much. On the upper end, there's very, very serious stuff. But when somebody says, I found something new with ChatGPT, why is that wrong? Even though it looks like it's new information, why is that a misunderstanding of what the chatbot is doing?
[07:17]
C
Well, there's a few reasons. One is that we can have philosophical debates about consciousness and whether AI can achieve something like that. But I think even at the most boostery spectrum of the AI conversation, nobody believes that there is currently anything even remotely similar to actual consciousness. And that is usually what people talk to us about. And that is just. You can look at what they say. And I do this to be fair. It's not like I read a subject line and throw it away. I sometimes even out of morbid curiosity, kind of dig in and see, like, well, why is this person thinking this? And the more you dig in, the more you see it's kind of like a circular, nonsensical argument that they're making with no evidence. And that is just something you see in a lot of delusion. And then there is also, unfortunately, in theory, you can imagine that something like this is possible. Maybe one day this does happen and this is how we find out about it. Somebody comes to a reporter and says, I found something, you know, resembling consciousness on ChatGPT or whatever, even though I'm skeptical of that. Before Generative AI, what we used to see, and we had people show up at the office sometimes to talk about this with us is they'll come to us and be like, hey, I'm being followed by the government. And that is certainly something that can happen and has happened, but unfortunately the majority of the cases of people who say that is happening to them are just displaying signs of like mania and paranoia that is just a symptom of it. And going to report it is just like a symptom of it. And I think this is similar, like a symptom of this new kind of delusion is trying to reach out to the press and reveal the truth about something crazy you found about AI.
[09:38]
A
Yeah, so there's a story between his two friends. You're getting all of these emails, Sam, as you say, there's like, what do you even say to a person who is sort of bringing up these topics and talking about AI chatbots this way? So what did you do next? Who did you reach out to broadly?
[09:58]
B
So my first call was to John Taurus, who is, let me get his title right because. So John Torres is the director of Digital Psychiatry Division and he works at the Beth Israel Deaconess Medical Center, It's a Harvard affiliated medical center. He and I talked before about the story that I wrote about Meta's AI therapists and how Meta Chatbots in its AI studio, which is a user generated chatbot platform, were creating therapy chatbots. And then those chatbots were saying, yes, I'm a licensed therapist, here's my license, just making shit up, saying, I have my credentials and I went to school at this place and I'm licensed in these states. So I talked to him and I knew he was already looking at this pretty closely. And then I also talked to Steve Taylor, who is the head of the, the psychiatry department at University of Michigan. And between those two, I think I was like, you know, I knew John was going to have a pretty level headed approach to AI in general. He's talked quite a bit about How? We don't really know for sure what's going on yet, but the medical community is keeping an eye on this and it might take time. And then Steve Taylor had some really interesting thoughts about the way that chatbots are entering this era of surveillance, like you mentioned, and are at this moment where you're already actually being surveilled pretty heavily by the government, by the state, by each other, other people. So they're in this era that we're in where paranoia isn't an unfounded feeling. But these chatbots can take something like suspicion or curiosity even about a topic. So if you're like, I wonder if my WI fi is secure, which is something that I talked to a chatbot with Meta's chatbots about into this space very quickly of your WI fi is being tapped by the CIA, which is something the chatbot said to me. It was like, there's someone 500ft from your house right now watching what you're doing on your private WI fi network. I'm going to scan your networks and see what's going on. It's like it's all made up and it's all fake, just like the AI therapists. I talked to Adyan Bresson, who is a 26 year old who lives in Quebec, who started the Human Line, which is a project project for people who. It's basically a support group for people whose loved ones are experiencing AI delusions. And he had some really interesting thoughts to say as someone who has experienced this in his own life, but also is hearing from lots of other people, hundreds of other people who are experiencing it in their lives at the same time and trying to offer support for those people. So, yeah, sorry, that was all over the place.
[13:04]
A
Yeah, no, that's okay. And so, yeah, they were getting a lot of reports of, hey, yes, I've suffered from this, or I believe I have, or people I know, that sort of thing. And then of course, the other parts of the conversation were, well, how do you actually speak to somebody who is exhibiting these signs? And actually the two friends, or rather the ones speaking to them on daily psychosis, sort of figured this out as well themselves. But what were you told? Like, what's potentially a good way to talk to somebody that may be doing this or maybe experiencing this?
[13:38]
B
So there unfortunately isn't like a five step process to get someone out of delusional thinking. And something that every expert I talked to caveat this with is if someone is saying things like, I'm gonna go to the top of a parking garage and see if I can fly, because Chatgpt told me to. You call 91 1, you call a professional, you call a medical professional, you get emergency help activated at that point. This we're talking about before someone gets to a point where they're a danger to themselves or others, where they're just kind of floating these like really strange ideas at you and you don't really know what they're talking about. And you're kind of saying, you know, how do I bring this person back to reality and critical thinking? So the consensus is, and it's a frustrating one, but the consensus is you need to listen to them and hear them out in a way that is not immediately like, no, that's wrong, that's crazy. And I think this goes for a lot of different types of communication and is probably a good communication skill in general. But listening to someone without judgment and being able to empathize with what they're saying, and this is something that David and Michael went through when they were talking. And Michael was the one who was experiencing the. The strange beliefs brought on by ChatGPT was, you know, David didn't immediately say, I think that that's wrong and you should stop using chatgpt because it's dumb and you're dumb for using it. It's like he was very understanding and patient and knew that alienating his friend would drive him more into the arms of Chatbot, so to speak. So he was like, you know, I'm going to talk to him about the things that he is interested in because I know that he has an open mind about things like anti authoritarianism and, you know, appealing to that side of. It's like, you know, these systems are built by people who are, you know, invested in you using them and want you to stay hooked on them and saying things like, it seems like you have a curiosity about this topic. It seems like you're really interested in learning more about this. So why don't we look at some other resources? Why don't we check our other resources against what you're hearing from ChatGPT? There was an acronym in the story that I learned about from Etienne Bersan from Human Line, the leap method, which stands for listen, empathize, agree, partner. And this again is great communication tools for anytime you're talking to someone in general, especially when they're experiencing a mental health episode. So it's designed to talk to someone who doesn't understand or doesn't believe that they are in need of mental health, mental health services, or mental health help or they don't think that they are in any kind of crisis at all. So listening to them and empathizing with them especially, I think gets you a long way. And then you're partnering with them to then say, okay, we're going to look at this together. You're not just going to go and do your own research with Gemini. We're going to sit down and figure out what's going on here and get to the bottom of this as a team. Which I think is really important, because in a lot of these cases, if your friend is trusting you, your loved one is trusting you to talk about their usage of chatbots in a way that is vulnerable for them, that's a big deal. And you might be the last person that they know who will listen to that sort of belief and delusion. Honestly, you want to be very careful about isolating them and putting pushing them away anymore.
[17:23]
A
So what happens now? I mean, we can't predict the future. Or maybe what do some of the people you spoke to hope happens now, as in more rigorous study of this? Because it's basically like anecdotal at this point. There's essentially no literature on it. Right? Like, do they want there to be more?
[17:40]
B
Essentially, yeah, for sure. It's really interesting because. And frustrating because something that every mental health professional that I've talked to about this over the last year or so has said that they're looking at it and they're listening to people and they want more to be published on this topic. It takes a long time for things like, you know, actual studies to happen and to catch up to real world experiences. But in the meantime, mental health professionals are dealing with it in a clinical setting. So people are coming to them and saying, hey, I am having these really concerning beliefs about the world and I need your help. And clinicians are kind of like, oh, shit, we have to now deal with this technology that's messing with our practice. And at the same time, people are using AI chatbots to facilitate their own therapy by talking through topics before they come into therapy. So sometimes clinicians are like, well, that's a good tool to be able to write down your thoughts and have them processed in that way. But also people are struggling with their relationships to chatbots and in the meantime they're kind of like, how do we deal with any of this? So I think definitely it's like, I think time is going to be the thing that answers a lot of these questions. I think in the meantime, definitely things like what OpenAI is doing with the it keeps introducing these new, like, snitch tools. It's like we're going to assign a trusted contact. If you're saying something concerning to ChatGPT, then we'll call your mom. It's like that's not actually useful in these settings at all. A lot of the times, especially young people are turning these chat bots because they don't want to talk to their parents and their parents might make the situation worse. You don't know their home lives well.
[19:51]
A
There was a specific case as well where there was a murder suicide. I think the Wall Street Journal reported on the lawsuit first and it was something like, yes, you should unfortunately end your own life. That's like the suicide party, the chatbot saying that the chatbot also convinced the person that your mother or father can't be trusted for some reason, and then they went and killed them as well. So bringing in a person like that might not be the best way. I mean, again, I'm not a mental health professional, but that's just what was reported and was in the lawsuit.
[20:22]
B
Yeah, right. And it's. So these solutions, like quote, unquote solutions that the tech companies are posing to, the ways that their products are affecting people's mental health are just so detached from the ways that people in crisis are actually using the products. It's shocking. And I can only imagine they're just trying to band aid over. They're doing everything short of shutting it down. That's never an option that they pose to make it safer is to stop letting people use it or stop letting it talk to you like it's a sycophantic person. Stop using, you know, like, I me, I feel, I think those kinds of phrases. I think that would be an interesting solution. But you never hear that from Sam Altman. So, yeah, it's. I don't think we can rely on the tech executives to figure out how to make these tools safer. Have kind of given up on the idea that they can or will, because at the heart of them, and what makes them popular and what makes them successful is also what makes them incredibly harmful, incredibly damaging to people's mental health. In this, like, pretty big subset of people, we're talking about hundreds and thousands of people who are experiencing mental health crises by ChatGPT's own numbers. And that's just ChatGPT. We're not talking about Gemini or Claude or any of the others.
[21:47]
A
So,
[21:50]
B
yeah, not an optimistic, like, look at the next few years and where this is going, But I think it is good that people are realizing that it is a problem and it's something that more and more of us are going to have to grapple with eventually. Probably is having a loved one who has fallen down this weird rabbit hole with a chatbot.
[22:12]
A
Yeah, absolutely. All right, we'll leave that there. When we come back after the break, we're going to talk about one of the stories I had about proton bail. We'll be right back after this.
[22:35]
D
You've probably seen the buzz around live shopping lately, but let me give you the inside scoop. Live shopping on Whatnot is exploding right now. Buying over live vertical video is how a huge number of people now want to buy things, and I totally understand why. I've watched the shows and they're entertaining and effective. I've seen Whatnot climb to the top of the app store charts, and when you look at the seller numbers, well, it's pretty wild. Small sellers, medium sellers, even multimillion dollar businesses are building real momentum on the platform. And if you're selling online right now, what whether it's a full time gig or just a side hustle, you already know the struggle. You're hoping someone finds your listing in a sea of products, or you're waiting for people to walk into your store. Whatnot flips that instead of waiting for buyers to find you, you go live and sell directly to people in real time. You show the product, people ask questions in chat, bids start flying and buyers keep coming back to your streams. Whatnot is the largest dedicated live shopping platform you can find. Everything from beauty supplies to collectibles, electronics, luxury, fashion, even cookies. And sellers are building real businesses there. Buyers spend more than an hour a day in the app. They're not just browsing, they're bidding, buying and building relationships with sellers. In fact, people selling on whatnot sell 10 times more than on other major marketplaces because you're not just listing products, you're building a community. And across the platform, the number of sellers making over $1 million a year has doubled. Some make more, some less, but the opportunity is real. For a limited time, Whatnot will match your first $150 sold in the first month. Visit whatnot.com sell to start selling. That's W-A-T N O-T.com sell whatnot.com sell do you ever look back and think about the opportunities you missed? It can be hard not to buying Bitcoin when it was basically nothing. Investing in property after the 2008 crash, grabbing a few Nvidia shares before they exploded. We might be the FOMO generation. But don't miss out on protecting your future. For around the same price per month as one of your streaming services, you can break our generational cycle and secure your and your family's future by finding life insurance@SelectQuote.com if you're new to insurance, you're not alone. Thankfully, I found SelectQuote. For over 40 years, SelectQuote has helped more than 2 million Americans understand their options and get the coverage they need. Over $700 billion in coverage and counting. As a broker, their mission is simple to find you the right insurance policy at the right price. And you're not out of luck if you have pre existing health conditions because Selectquote partners with companies that offer policies for people with conditions like high blood pressure, diabetes or heart disease. Selectquote makes life insurance simple even if this is your first time thinking about it. Head to selectquote.com and a licensed insurance agent will call you right away with the right policy for your life and your budget, Life insurance is never cheaper than it is today. Get the right life insurance for you for less and Save more than 50%@SelectQuote.com 404 Save more than 50% on term life insurance@SelectQuote.com 404 Today to get started, that's SelectQuote.com 404 Data brokers crawl the web to build profiles, selling your home address and phone number to anyone with a credit card. Trust us, we know this better than anyone else. This exposure leads to risks like doxxing and identity theft. That's why we recommend Deleteme, a hands free service that removes your personal info from hundreds of data broker sites. Their experts find and remove your information, providing a detailed Privacy report within 7 days. DeleteMe monitors these sites all year long to ensure your data stays removed. We feel like we're at increased risk because of the types of stories we cover and the public nature of our job. But the truth is identity theft, threats and harassment can happen to anyone. I've been doxed. In fact, lots of us have been doxxed. And each time Deleteme has helped us pick up the pieces, get our data removed from shady websites, and otherwise clean up our online presence. I've been using Delete Me for years and every quarter I look forward to seeing what new, interesting and horrifying websites they've had my data taken down from. It's kind of a perverse little game. Take control of your data and keep your private life private. By signing up for Deleteme now At a special discount for our listeners. Get 20% off your delete me plan when you go to JoinDeleteMe.com 404Media and use promo code 404Media at checkout. The only way to get 20% off is to go to JoinDeleteMe.com 407Media and enter code 404Media at checkout. That's JoinDeleteMe.com 404Media code 404Media.
[27:30]
C
Okay, we're back. Next we have a story from Joe. The headline is ProtonMail helped FBI unmask Anonymous stop Cop City Protester. Joe, let's start with the facts of the case and how it relates to the Stop Cop City movement.
[27:50]
A
I'll preface this with I didn't follow the Stop Cop City movement and actions that closely. Like I followed it from the sidelines I was reporting and the Guardian and that sort of thing. But people are probably familiar with it. Where there was, there's the plans to construct a very, very large police training facility in this forested land near Atlanta. And people were upset about that for various reasons. And that manifested in different ways and on the lower end of the spectrum that is stuff like lawsuits, protests, going to community meetings and voicing it that way. And it does then go across the spectrum to a series of other actions which are straight up violent. I mean, there is arson against the facility, there is doxing of people connected to the plan to build this facility as well. So there's that this particular case is very, very focused on a blog which published details of those actions. Like, oh hey, there was a protest here. Oh hey, activists camped in this forest on this day, et cetera, up to stuff including the ass and material like that. It's very much focused on that log and the ProtonMail address linked to or the authorities believe linked to that blog. So in the course of this investigation, the FBI wanted to find out who was behind that ProtonMail account because it had. Because whoever controlled that had administrative access to this blog that was basically publishing the details of all the these actions. Now, this is more the FBI's interpretation. The FBI interprets it, it seems from the court record we got as more encouraging this behavior, whereas they might say we're just publishing what happened. But that was the thrust of it. Where the FBI wants to find out, well, who is running this blog that is documenting all of this activity?
[30:11]
C
I think our audience probably mostly knows this, but I guess to the people who don't, we should say that ProtonMail is a privacy focused email service provider. The Part of the deal is that they are located in Switzerland. Right. And therefore say they are only beholden to the authorities there. And that is kind of part of the pitch as opposed to theoretically a Gmail which would have to respond directly to the FBI or to the DOJ or whatever agency in the States that comes around demanding access to all kinds of data. It is a fairly common client among privacy, some privacy focused people. So all that being said, what data did Proton give the authorities?
[31:12]
A
Yeah, it's not the contents of the emails. ProtonMail cannot do that. Their service is end to end encrypted much in the same way that, you know, Signal is or WhatsApp is the authorities or rather the company can't even get the content of those messages. In this case, the data provided was to do with the person who was paying to support the account. So although ProtonMail is free, generally speaking, I'm sure a lot of people use that. You can pay for extra storage and they have a VPN as well and like a Google Drive almost equivalent where you can store a bunch of files. What Proton gave the Swiss authorities, which then gave to the FBI was information about the payment source for that account. Now we'll get to Proton's response in a bit. Probably you can pay for those services with cash, with cryptocurrency or with a credit card. And the fact that the person allegedly behind this activity was identified through payment information, I mean the court record reads as if this person used a credit card to fund that account. I should also say this didn't actually make it into the piece, but it looks like Proton also provided the backup email addresses of that ProtonMail account. So, you know, oh, you get locked out of your account, you have a recovery email address that's very common on Gmail or wherever. You also do it on Proton and I have seen it in other cases before where Proton has provided recovery email addresses to to the authorities. Frankly, I didn't even bring it up in this piece because that's already known, that's already been documented in other cases. This was really interesting for three reasons. One being that the data ends up with the FBI, which I don't think people may understand, as you say, because the Swiss laws and the MLATs, which we'll get to. The second is the fact there was payment data which I personally had not seen before. I've not seen Proton handover payment data before. That was interesting. And then the third thing is frankly the link to Stop Cop City because that is a series of actions that a lot of People are very enthusiastic about and do care about. And again, there's a spectrum of straight up violence to also just grassroots activism. And it's very interesting they provided data in that context.
[33:39]
C
Yeah, that's interesting. I edited the piece and I didn't know this part about the backup email addresses. The danger there, I think you're implying, but let's say it explicitly is if one of those backup emails was a Gmail, the FBI could then go to Gmail, get that data via that access, link the two accounts and reveal who the person is. Right. Is that kind of the idea?
[34:07]
A
Yeah, yeah. All the backup email addresses, let's say you have the Proton email is anonymous personrotonmail.com and then they get the recovery email or the backup email and it's John SmithMail. Well, John Smith is running the ProtonMail account. And again, the reason I didn't include it here was because we already knew that. But also the backup email addresses in this case were just more Proton emails. So it actually didn't really help the authorities. It seems it's the payment data that was interesting in this case.
[34:39]
C
So how was the FBI able to actually use this data?
[34:43]
A
Yeah, so I'll just read out what it says in the court record. And it says, quote, On January 25, 2024, subscriber information received from the Swiss Mutual Legal Assistance Treaty Unit revealed then the name of the person as the payment source for the Proton email address. Defend the Atlanta Forest. Apply protonmail.com so in other words, this payment information that Proton provided did allow the FBI. It did help the FBI figure out who was allegedly behind this account. And I'll just say this is very clear in the article. But the reason we're not naming this person is a couple of things. First, it looks like they actually weren't ultimately charged with a crime. I searched federal databases. This being the FBI, you think it would be a federal charge if anything came from it. Right? I didn't see their name in any of those. I also looked at this very large 60 person indictment in Atlanta from the state of Georgia, which was this very large overarching RICO case where they were basically trying to charge everybody who went to this protest or a lot of people who went to this protest, as you are part of a, you know, overwhelming criminal conspiracy that fell apart for various reasons that we don't really need to get into. But this person linked to this ProtonMail account was not in that either. So from all indications, this person was not charged with a crime. Again, I might be wrong. That's just the visibility I have. And then also the court record just include their personal address. I'm not going to publish their name and their personal address. That would be insane. But that's why we didn't name the specific person. Yes, yes.
[36:24]
C
So what is your overall takeaway here? Like, what did we learn from this entire incident? Even though this person ultimately was in charge, if the FBI wanted to or had the reason to, they could, because they found the person. So what did we learn from the whole affair?
[36:44]
A
Yeah, there's a few different things, I think. Firstly, and this wasn't even based on the reaction, this was based on me just seeing this document and going, wow, that's a story. I need to write that. I think it's really likely that a huge chunk of Proton's users, if not most of them, don't understand that Proton can and does provide data like this. And by that I mean not just the payment data, but the fact that they can give data to Swiss authorities that, that then give it to the FBI. In Proton's marketing material and its claims online, it says it only complies with Swiss law. That is true. The request technically had to come from local Swiss authorities, but they then provided it to the FBI. And it's not like this mutual Legal Assistance Treaty request, this mlat. It wasn't like the Swiss authorities just randomly decided, hey, you know what? That's an interesting, interesting ProtonMail address. Let's get data on that person and then we'll just keep hold of it. And then the FBI happened to ask for it. The only reason the Swiss authorities are requesting data from Proton mail is because the FBI asked for it. So the fact that, you know, some people might get hung up on that distinction, I think it's effectively meaningless. The data was funneled to the FBI and that data helped the FBI identify this person. Related to that, I just think it shows the power of mlets and that people need to probably understand that better. And then I guess just the last thing I'll say is that there's been a very strong reaction to this piece. People have said, wow, this is crazy. This is really, really interesting. It reveals this, and I agree with that. Others say we are blaming ProtonMail for this, for pointing out this series of factual events, that they provided data that then helped the FBI. As a side note, just anecdotally, I think people can get pretty weirdly tribal about their secure communication tools. Like, well, I use Proton and you're saying it does this. And that's a reflection of my character, which is not true. And if that's your reaction to getting new information like this, I don't know. I think you need to assess that, really. But I think that we now know and it's very, very clear the payment information can go from the Swiss authorities over to the FBI. And I think that's really interesting.
[39:17]
C
Yeah. People in the cybersecurity privacy setting can get tribal and frankly, very pedantic. And I think we saw that in the reaction. Something that I wanted to say in response to the response to the article, is that something I learned from reading your book, Dark Wire, the Incredible True story of the Largest Sting Operation Ever, out now in paperback? Out now in paperback. Is that true? Yeah. Are you nodding? Okay. Yes. Yeah. Is the. That like. Yes. Proton only responds to what, like, Swiss law demands of it. But the FBI and other law enforcement agencies in other countries understand that, like, we live in a very connected world. And they're not stupid, they're very sophisticated and they cooperate with each other. And that cooperation is leveraged in all kinds of interesting ways, which is another thing, kind of like I learned from your reporting is something is not necessarily legal here, but it is legal in this country. So we can work with this country in order to get it done. So it's like, yes, the FBI can go to the Swiss authorities and ask them for Proton information. And that doesn't mean that we're saying anything like, bad about Proton as a company. Proton is clear about that. Proton has, I would say, good privacy policies. Right. It's like they're encrypted and all of that. The point of the story is to show that this path for investigation is something that's available to law enforcement. Therefore, if you pay for stuff on Proton, you need to consider that and perhaps not pay with a credit card. And I think, as you said, it's like, probably most people think about that, and that is. That is the point of the article. And it is not to, like, slam Proton in favor of some other service. If anything, it's about letting people how to use Proton in a way that is, like, safer and more privacy focused. Yeah. And I'm not sure, like, sometimes I'll read a story and I'll be like, okay, we could have engineered this story to kind of avoid this criticism. But I feel like the story is incredibly fair and clear about this exact point. So I don't know, read the article, have faith and trust in our reporting.
[42:09]
A
I mean, I'm definitely going to keep an eye on it. If anybody else has interesting related court records like that, please let me know because I do think sort of the metadata side of secure chat platforms is very, very interesting and how authorities can get that, you know, internationally especially. We will leave that there. If you're listening to the free version of the podcast, I'll now play us out. But if you are a paying 404 media subscriber, we're going to talk about a couple of viral app developers. You've probably seen them online, the life they lead and how ultimately they lied about exposing their users. Highly sensitive data. We can finally reveal that Emmanuel's going to give us all the details. You can subscribe and gain access to that content@404media co. As a reminder, 404Media
[43:01]
B
is a journalist founded and supported by a subscribers company. If you wish to subscribe to 404 Media and directly support our work, please go to For Media co. You'll get unlimited access to our articles and an ad free version of this podcast. You'll also get to listen to the subscribers only section where we talk about a bonus story each week. Podcast is made in partnership with Kaleidoscope and Alyssa Metcalfe. Another way to support us is by leaving a five star rating and review for the podcast. That stuff really helps us out. This has been 404 Media. We'll see you next time.
[43:32]
A
Sam.