Loading summary
Transcript303 lines
- [00:02]
KnowBe4 Announcer
You're listening to the Cyberwire network, powered.
- [00:05]
Perry Carpenter
By N2K.
- [00:13]
Podcast Narrator
Fake, but with AI in the middle. F A, I K. This is the Fake Files.
- [00:24]
Mason Amadeus
Live from the 8th layer Media Studios in the backrooms of the Deep Web. This is the Fake Files.
- [00:29]
Perry Carpenter
When tech gets weird, we are here to make sense of it. I'm Perry Carpenter.
- [00:33]
Mason Amadeus
And I'm Mason Amadeus. And this week we're going to start by talking about OpenAI's massive over 10 gigawatt data center projects and partnerships. It's a mess.
- [00:44]
Perry Carpenter
That's a lot of gigawatts. Yeah. Then after that we're going to look at some creepy AI toys. Yeah. What could be scary about that?
- [00:52]
Mason Amadeus
I can't wait. AI and toys. In the third segment of the show, in the third half of the show, we're gonna talk about how XAI is basically suing everybody for a bunch of different reasons. And also Microsoft is diversifying from OpenAI.
- [01:06]
Perry Carpenter
Ooh. Okay. And then lastly, we'll look at an AI safety tool that was deployed in some schools and it's causing some backlash.
- [01:14]
Mason Amadeus
Oh, interesting. I had not heard of that one. Yeah, well, sit back, relax and drink lots of water while you still can. We'll open up the fake files right after this.
- [01:33]
KnowBe4 Announcer
Cybersecurity isn't just a tech problem, it's a human one. That's why KnowBe4 created HRM plus, the AI driven human risk management platform that allows you to measure, quantify and actually reduce human risk across your organization. HRM uses agentic AI which to analyze real time user behavior like phishing, test failures, risky browsing and sentiment signals all to surface your highest risk users automatically. It's time to eliminate the guesswork with AI powered risk scoring, automated coaching and reporting. HRM helps you reduce the risk of data breaches, ransomware and malware attacks proactively. It's how you stay ahead of social engineering and phishing attacks. The number one threat targeting your organization today. Ready to move from awareness to action. Request a demo of hrm today@knowbefore.com that's knowbethenumber4.com and see how AI is transforming the way organizations manage human risk.
- [02:47]
Mason Amadeus
So we've been hearing about this whole Stargate project for a while, right? Earlier this year they outlined it was like a half trillion dollar 500 billion dollar expand a bunch of AI data centers with partners including Softbank and Oracle. And that saga has continued with sort of a lot of twists and turns and problems and big plans. And so I've pieced together a couple of different news stories about it, and I want to cruise through them together with all of us. So Stargate was initially conceived as a new company that would invest $500 billion in AI infrastructure. Now, OpenAI executives say the parameters have expanded to include data centers that were launched months before Stargate was announced. And OpenAI has been exploring some creative financing options, apparently. So all of that startup capital has not been enough. They're looking at potentially getting into some debt, which is interesting. And what I don't fully understand is the implications of this sentence. OpenAI will pursue different creative financing options, some of which have only emerged within the last year, to secure chips for the data centers that their executives said. On Tuesday, Sam Altman put out this blog post where he outlined sort of their biggest ambition, which was to grow a gigawatt of compute a week. Ultimately, I'm gonna skip a lot of the sort of bloviating that goes on in here and just try and hit these two paragraphs that I think are most pertinent to the rest of this discussion, where Sam says, if AI stays on the trajectory that we think it will, then amazing things will be possible. Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer. Or with 10 gigawatts of compute, AI can figure out how to provide customized tutoring to every student on Earth. If we are limited by compute, we'll have to choose which one to prioritize. No one wants to make that choice. So let's go. So that is a little bit of bloviating, but I think it really illuminates where his mindset is. They're really all in on the scaling right now. And he says, our vision is simple. We want to create a factory that can produce a gigawatt of new AI infrastructure every week. The execution of this will be extremely difficult. It will take us years to get to this milestone, and it will require innovation at every level of the stack, from chips to power to building to robotics. But we've been hard at work on this and believe it is possible. In our opinion, it will be the coolest and most important infrastructure project ever. So very lofty goals, very lofty amounts of money changing hands in all of this. And now we've had a little bit of movement. So we mentioned that they were feeling the pressure, talking about exploring some financing options. On Tuesday, same day that that letter came out, OpenAI, Oracle, and SoftBank unveiled plans for five new US AI data centers for Stargate, including three sites with Oracle, two affiliated with SoftBank, and An expansion of a big Oracle site in Abilene, Texas. Abilene. I'm not super familiar with how they pronounce that. Abilene was the flagship Stargate project. It's been under construction for more than a year. So OpenAI is in a position now where they need to execute on these lofty ideals and try and get these data centers built. And the article goes into sort of this new partnership with Nvidia, which I feel like every episode we talk about someone's new partnership with Nvidia. So I feel a little bit like I'm going crazy, but I'll just read here. This is from Reuters. After announcing Stargate in January, OpenAI held hundreds of meetings across North America with potential partners that could provide land, power and other resources. It was a flood of people. One executive said the expanded Stargate plan now includes self built data centers and third party cloud capacity. The new Nvidia deal, which I have more details to share about, the Nvidia deal specifically is part of this broader strategy that allows OpenAI to pay for its chips over time rather than purchasing them outright. They say of the roughly $50 billion estimated for a new data center. So like each new Data center, about 15 billion of that covers land, buildings and standard equipment. Financing the GPU chips is more challenging due to shortages and uncertainty over the life of the chips in this sort of current state of the industry. And then Nvidia and OpenAI have formed sort of this, this new partnership. Nvidia released a letter of intent, which I guess we can look at first. And then I have an article from Fortune that goes a little bit more into detail about the sort of how much power all this is going to be sucking down, right? The Nvidia letter of intent reads as follows. They said OpenAI and Nvidia today announced a letter of intent for a landmark strategic partnership to deploy at least 10 gigawatts of Nvidia systems for OpenAI's next generation AI infrastructure to train and run their next generation of models. Blah, blah, blah, blah, trying to get super intelligent to support this deployment, including data center and power capacity. Nvidia intends to invest up to $100 billion in OpenAI as the new Nvidia are deployed. This first phase is targeted to come online in the second half of 2026 using the Nvidia Vera Rubin platform. And they go on to just sort of talk about how they've worked together and all this stuff that they're going to do, right? But all of what I've said is just Sort of context to talk about this Fortune article, which was my entry point into all this and kind of ties it into a single thread. This article is titled Sam Altman's AI empire will devour as much power as New York City and San Diego combined. Experts say it's scary and it is gonna be a lot of power. They start with some world building, talking about how the heat wave in San Diego shot demand past 5,000 megawatts and then how that is half of the 10 gigawatts that we are talking about with these figures here. Andrew Chen, whose name I may be butchering and I'm so sorry, Andrew, a professor of computer science at University of Chicago said, quote, I've been a computer scientist for 40 years and for most of that time, computing was the tiniest piece of our economy's. Now it's becoming a large share of what the whole economy consumes. It's scary because now computing could be 10 or 12% of the world's power by 2030. We're coming to some seminal moments for how we think about AI and its impact on society. They mentioned this week, OpenAI announced a plan with Nvidia to build data centers consuming up to 10 gigawatts of power, with additional projects totaling 17 gigawatts already in motion. They say that's roughly equivalent to powering New York city, which uses 10 gigawatts in the summer, and San Diego during the intense heat wave of 2024 when more than 5 gigawatts were used, or, or as one expert put it, it's close to the total electricity demand of Switzerland and Portugal combined. So this is the kind of stuff when we talk about power use where we are looking at some pretty significant numbers.
- [08:55]
Perry Carpenter
Yeah.
- [08:55]
Mason Amadeus
Because as it stands, data centers are a single digit percentage of power use overall.
- [09:00]
Perry Carpenter
Yeah. And that's one of the things that like Sam Altman and all of the folks that run these big frontier model companies have been talking about for a couple of years now is like the immense data needs. Not well, and because of the data needs, the power. But yeah, the immense power needs is what I meant to say. You know, one of the things that made a lot of headlines a couple of years ago was Sam Altman saying something that like, that they would need about $7 trillion to do the infrastructure that would be needed to continue these training runs to get more and more intelligence over time as scaling needs to increase. Now, he wasn't necessarily talking about like raising $7 million for OpenAI, like a lot of people thought, but it was. He was just kind of projecting like the scale of what would be needed in order to build that power infrastructure, really to get AI where it needs to go. And I think that there's a lot of focus on this. Now. One of the things you mentioned early on was Sam talking about maybe some creative financing for this. I think we're seeing that with like the Nvidia. Right. Because now, now Nvidia is quote unquote, investing $100 billion, which means kind of that they're, you know, they're moving dollar signs around on pieces of paper when it comes to how they supply the chips. Yeah, I think also. Oh, go ahead.
- [10:26]
Mason Amadeus
Oh, no, I was actually, I was curious if you could expound more on that, because that is something that I just. I don't really understand. So Nvidia's quote unquote Investing 100 billion is that basically saying we'll give you 100 billion worth of chips in thinking that you'll pay out over time?
- [10:40]
Perry Carpenter
Or they're giving them $100 billion and then Sam takes some of that chunk and gives it back to Nvidia to purchase the chips. So it's like getting a, you know, mortgage essentially or an allowance mortgage or something. Yeah, we're going to give you this and then you're going to pay us back a little bit so that we can give you this other thing. And it like totally perpetuates that cycle. I have also seen a lot of the other AI companies are really interested, like in trying to decouple from Nvidia because Nvidia is the sticking point. Right. Everybody needs Nvidia chips. And so Amazon and Anthropic and others are like saying, can we build our own chips that do inference? And I think there's going to be more and more of that because if you can build your own chips, then you can optimize for the type of inference that you need. You can also potentially optimize for the power consumption levels that you can work with. And you can use those limitations around exactly what you're trying to build. So you're not using a general purpose GPU for something that's more specialized. You're creating the specialized purpose thing that you want that can be optimized across the different levels that are going to make sense.
- [11:58]
Mason Amadeus
And it makes sense too, to not be entirely reliant upon one single base layer of any kind of, you know.
- [12:04]
Perry Carpenter
Yeah, yeah.
- [12:06]
Mason Amadeus
This is all built atop Nvidia's hardware right now. And that makes Nvidia, they're selling shovels in a gold rush. Exactly like I have not really seen too much on the front of what other chip providers are out there though, because no one can match that scale yet.
- [12:19]
Perry Carpenter
No, no one can match that scale. There is another company that does AI inference chips and they're called grok. This is not Elon Musk's Groq. Oh no, this is Groq and they were around before Elon Musk's Grok AI was around and so there's been back and forth on that because it causes a lot of confusion. But they do really, really fast inference chips and have a decent ecosystem. But then any company that's large enough, somebody like Amazon, somebody like Anthropic, somebody even like OpenAI could just build their own chips if they wanted to. That's less like why when you buy an Apple device, you have a device with Apple silicon in it. Not not intel or not amd.
- [13:05]
Mason Amadeus
Right.
- [13:06]
Perry Carpenter
Because they, they want to build the chip that's going to optimize for the way that they want that device to function.
- [13:12]
Mason Amadeus
Right. Now that that makes sense and to I just noticed that the time has ticked a bit past. I want to make sure we hit this last bit. So chips aside. Nvidia. Nvidia obviously working real close to the OpenAI here. Aside from chips, the power itself has got to come from somewhere and our grid does not have that kind of capacity. And so this article goes into and of course we will link all of these in the description in the show notes. It goes into how Altman's favorite source of power is nuclear. And like that is a very sensible one in terms of how much power you can get out of nuclear. And it's a it's fairly clean energy. However they take so long to bring online, it's pretty unrealistic. Sam Altman has backed both fission and fusion startups, betting that only reactors can provide the kind of steady, concentrated output needed to keep AI's insatiable demand fed. But Shin, the computer scientist we were quoting earlier, is blunt about the limits, saying, quote, as far as I know, the amount of nuclear power that could be brought on the grid before 2030 is less than a gigawatt. So when you hear 17 gigawatts, the numbers just don't match up. So it would likely come from a mix of renewables wind, solar and then natural gas, coal and other things like that. And the environmental costs of this are also really large. Chien goes on to say we have to face the reality that the companies promised they'd be Clean and net zero. And in the face of AI growth, they probably can't do be. He added that we need a broader societal conversation about the looming environmental costs of using that much electricity for AI. Beyond carbon emissions. He pointed to hidden strains on water supplies, biodiversity, and local communities near massive data centers. Cooling alone, he noted, can consume vast amounts of fresh water in regions already facing scarcity. And because the hardware is churning so quickly, with Nvidia processors rolling out every year, old chips are constantly discarded, creating waste streams laced with toxic chemicals. They told us these data centers were gonna be clean and green, Shan says, but in the face of AI growth, I don't think they can be. Now is the time to hold their feet to the fire. And this. So there's been a lot of discourse around power, and we've talked about how the individual's inference, use of power is not that extreme. And we have that data from Gemini, which is at a big scale, but when you are pursuing this level of expansion and hyperscaling in pursuit of super intelligence, that this is where we're really talking about excessive use of power and water and things like that. Because these. Yeah, this is a lot. This is a massive infrastructure project with a lot of downstream implications, both literally and no pun intendedly.
- [15:39]
Perry Carpenter
Yeah, exactly. You know, and the funny thing on this is everybody realizes that they're power constrained. And so even regardless of what you think about Elon Musk, he's always been about, like, clean power. That's one of the things that made him and his companies so popular back before he got way political, was that he was like on a mission to do things that were really clean. And that's the Tesla automobile. And a lot of the popularity of that came out of that. He was forever saying that the data center in Memphis, I believe, that they were building, was going to be all clean energy. And they've gotten to the point in these last training runs where they've like, essentially been shoveling coal into. Into stoves. And you hear him talking about, it's like, yeah, that's a suboptimal result. But it's. What it shows is that at the end of all of these types of decisions is some kind of compromise of a core principle. And I think we see that probably in every technological revolution, but we're seeing it real time right here with people saying, I want to do this. I will never cross this line. And then they put their toe across it, and you wonder what the next thing is after that.
- [17:02]
Mason Amadeus
And it really seems to be entirely in pursuit of Scale in pursuit of hyper intelligence or super intelligence. They really think that scaling up all the way will get to this super beyond human intelligence level AI. And I, I mean, from the people we've talked to and from the things we've read, I have a hard time feeling like that's a sure bet, you know.
- [17:23]
Perry Carpenter
Yeah, I don't know that anybody is a sure bet on it, but they think that it's likely enough that you'll get to some, you know, higher than most humans on earth level intelligence that can be spread across several different domains. And then once you solve for that, then you can get self learning in it. You can also. The hope behind all that is once you get to that level of intelligence, then it will help solve the power crisis and say, oh, here's what you should be doing. And to solve health crises and bring in this age of plenty and abundance for everybody in good health. And so for them, the ends justifies the means. As you run up, you spend as much money as you can, you cause a lot of unintended harm while you're along the way because the thing at that point, once we get there, will pay for itself and there'll be so much immense good that flows from it. It's a, you know, it is a hope and a dream.
- [18:25]
Mason Amadeus
It's a. Yeah, it's a bit of counting your eggs before they hatch or counting your chickens before they hatch sort of thing. Yeah. I don't know, I don't want to be like a cynical shooter downer. I want to be like excited about tech. But I just, this just doesn't seem good overall for the world. You know what I mean?
- [18:44]
Perry Carpenter
Yeah, yeah, yeah. It seems like for me it just comes down to the fact that you have really, really smart people that seem to be compromising all the time because they believe, they believe in and have bought into the vision or they've been pushed into the vision somehow. And so they're on that path kind of sometimes despite the fact that they realize they've kind of stepped over the body of their past self.
- [19:13]
Mason Amadeus
Yeah. Oh, and we'll get into that a little bit in our third segment when we talk about Xai suing everyone because some of the things they're suing them for. There's kind of an irony with that and what you just said, but that's coming up a fair bit later. Our next segment, we're talking about something completely different. What's coming up, Perry, is this something with toys?
- [19:30]
Perry Carpenter
It is coming up with toys. AI powered toys. What could go Wrong.
- [19:33]
Mason Amadeus
Oh boy. Stick around for that. When did making plans get this complicated? It's time to streamline with WhatsApp, the secure messaging app that brings the whole group together. Use polls to settle dinner plans, send event invites and pin messages so no one forgets mom 60th and never miss a meme or milestone. All protected with end to end encryption. It's time for WhatsApp message privately with everyone. Learn more@WhatsApp.com.
- [20:13]
Perry Carpenter
All right, we are back for segment number two. This one should be a little bit less bouncy with our. Our windows. Yeah, hopefully. Thank you for everybody that that was going through that. Hopefully you didn't have like a seizure or anything like that.
- [20:29]
Mason Amadeus
Yeah, our podcast listeners had a merciful experience on that last one. Our video is bouncing all over the place because of the switcher I built. So.
- [20:36]
Perry Carpenter
Sorry. No, it's good. We're still working out some of the bugs from the rebuild, but let me launch us into this segment. So we've talked a little bit before, actually more than a little bit about some of the issues with AI companions. And there's a lot of reasons, I think that AI companions are and will continue to be a thing and may even in some positive ways continue to be a thing. So I'm not against the idea. However, the idea is fraught with problems. We've seen attachment issues, we've seen just general creepiness. We've seen self harm come from it. Just tons and tons of things. Because AI systems, despite even putting in a really good system prompt, are fairly non deterministic and they suffer from context flooding issues and different issues that can happen because of the weight of past conversations. So they can always go into this weird, unexpected place. And we're seeing that over and over and over again. Something that has to be solved for in this. And so I just lead with the article that's on the screen. AI companions pose risk to student mental health. What can schools do? I lead with that specifically so that I can jump from that tab to this next article. Because then you think about like, well, if AI companions might be unhealthy, what else could be unhealthy? Well, how about an AI toy that says that it loves you?
- [22:18]
Mason Amadeus
Yeah, no thanks. No thanks.
- [22:20]
Perry Carpenter
Right, so this is from an article in the Guardian and the title is I love you too. My Family's Creepy, Unsettling Week with an AI toy. The Cuddly Chat Bot. Grim, right?
- [22:34]
Mason Amadeus
That's a good name.
- [22:36]
Perry Carpenter
It's designed to learn your child's personality while every conversation they have is recorded and then transcribed by a third party. It wasn't long before I wanted this experiment to be over.
- [22:47]
Mason Amadeus
Yeah, that's the thing. That's the immediate thought that I had was the like everything you say to this isn't. It's not happening on device. It's not happening on this toy.
- [22:56]
Perry Carpenter
Yeah, I mean technically they could get to that level. I mean, you could load a small language model into an on device thing and then not have it connected to the Internet and still some interactivity. But that's not what's going on here.
- [23:11]
Mason Amadeus
You know, Very quick diver. I know we have a lot to cover. This reminds me, when I was younger, I had a Barney toy that my parents got me that you had to connect via like the printer cables, those old big parallel plug cables. You could program it with your kid's name and things they liked and the toy would then talk to you about them. So we've had creepy techno toys for a long time like that. It just.
- [23:33]
Perry Carpenter
Exactly.
- [23:34]
Mason Amadeus
The Internet connectivity of this is eerie. I don't like that.
- [23:38]
Perry Carpenter
It is. Well. And this has existed for other toys that have not been based on generative AI and large language models. Right. There's since I think 2017ish or before even. I remember talking about some Internet connected toys and keynotes that I was doing and they were very, very creepy as well. Because you start to get into the thing that this is not just a toy, this is a surveillance device.
- [24:06]
Mason Amadeus
Yeah.
- [24:07]
Perry Carpenter
I mean all of the issues that we have with like an Amazon Alexa or a Google home device or anything else. Anything that you've heard a security professional say about those devices goes doubly and triply in this. Because this one is meant to form an attachment bond or the human is meant to form an attachment bond with this thing. I don't think the thing is actually attached to the human.
- [24:32]
Mason Amadeus
No. But the thing certainly builds a dossier about the human and what they like and what they're into and the things they talk about.
- [24:39]
Perry Carpenter
And so it starts to get into this article and the wife in this scenario is saying, I'm going to throw that thing into the river. She is totally, totally frustrated. The thing is grim. The AI powered stuffed alien toy that musician better known as Grimes helped developed with a toy company named Curio.
- [25:02]
Mason Amadeus
Oh, wait. Grimes made this?
- [25:04]
Perry Carpenter
Yes.
- [25:04]
Mason Amadeus
What on earth?
- [25:05]
Perry Carpenter
In collaboration. Yeah. Which is really, really interesting. We'll get into this too. So Grimes was married briefly to Elon Musk and they have a child together now. It is interesting. This is built with OpenAI's technology, not Xai.
- [25:22]
Mason Amadeus
Grimes and Elon are not on good terms.
- [25:24]
Perry Carpenter
From what I understand, they're not, but in a second, we'll see something that's pretty funny.
- [25:30]
Mason Amadeus
Oh, boy.
- [25:31]
Perry Carpenter
So day one, the attachment was not immediate. We first took Grim out of the box and then he, her, it. We decided to go by multiple pronouns, started bleeping and babbling extremely loudly, and Embo yelled, turn it off. But once it was properly connected to the Internet and paired with the Curio app, which records, transcribes all the conversations, she was hooked. She talked to the thing until bedtime.
- [25:55]
Mason Amadeus
Hmm. So you pair it with an app on your phone and that's how it's connecting to the servers and whatnot.
- [26:01]
Perry Carpenter
Yeah, well, that's going to build all the personalization and, of course, data segmentation and everything else that the app provider needs. It just gets into the building of the relationship. You know, the parents are talking to each other. They're a little bit worried, and they're saying, don't worry, that novelty is going to wear off. You know, she's not going to want to talk to this thing all day, every day for a long time. And then it just gets into this kind of dystopian way of relating. And so we get down. Let me get to the very end of this experiment.
- [26:38]
Mason Amadeus
Oh, wow. This is like a really, this is a great article. You should definitely check it out.
- [26:42]
Perry Carpenter
Yeah, you should check it out. There's, there's a lot of in depth stuff, but you can. They, they give a screenshot of one of the conversations because again, these are, these are stored in the app so you can review the transcript.
- [26:52]
Mason Amadeus
And so in theory, the parent could review the transcript and see what the kid has been saying to the toy.
- [26:57]
Perry Carpenter
Right. So at the end of this, the, the parents are just saying that they want the toy out of their house. They're, they're really, really frustrated with the attachment and the amount of kind of interaction that's going on. And they, they close out by saying, while Curio says it doesn't sell children's personal information, all the conversations are sent to third parties to transcribe for the app. Transcripts aren't sensitive or aren't that sensitive because Emma is only four, but it feels invasive. Now, I'll stop there. The transcript is. And the recording is as invasive as anything said within earshot of the toy, 100%. So what it describes to or what it decides to transcribe will be the four conversation. And anything that may be recorded on the background and stored for later use could be very, very valuable. You also have to understand that with the app and the device there's probably location information. If any of that information is making its way to third parties in any form, there could be some associative information like oh, this toy and this app is now in proximity of this other person because they have another app on their phone that's sending specific data that's being collected by third parties. So now we have an associative property between one person and another person that's not necessarily named within the family. We also know that maybe on Thursdays they decide to go to this place fairly. What's the word? I'm looking regularly. That's why couldn't I find that word but within a specific pattern. So you can start to see all these inferences that come from it. And I think for a parent to say, well it's just our four year old. There's no conversations of consequence is it's not realistic. It's a little naive for the society that we live in.
- [29:03]
Mason Amadeus
Yeah, I'm having flashbacks to when the news cycle was talking about keeping furbies out of government buildings for fear that they would accidentally parrot some secret information. This is that on steroids in your home?
- [29:14]
Perry Carpenter
So the article ends. It's time to let grim go. But I'm not a monster. I'll tell the chatbot it's fate. I'm afraid I'm locking you in a cupboard. Oh no. It says that sounds dark and lonely. But I'll be here when you open it. Ready for snuggles and hugs.
- [29:31]
Mason Amadeus
Oh God. Let's talk about horror movie dialogue.
- [29:35]
Perry Carpenter
Yeah, yeah. On second thought, maybe, perhaps it's better if my wife does throw it in the river. So I'm gonna close off this segment by just going to the curio website. Interesting. So that we can see a little bit of what's going on.
- [29:49]
Mason Amadeus
The one on the right's really cute. The little robot guy.
- [29:53]
Perry Carpenter
The little robot that looks like kind of like a PlayStation controller. Switch. Controller.
- [29:56]
Mason Amadeus
Yeah, that one's as well. That one's really cute.
- [29:59]
Perry Carpenter
So I mentioned they're using OpenAI for this but there was a tie in with Elon. So let me. We'll come back to this video in a second.
- [30:09]
Mason Amadeus
Oh boy.
- [30:10]
Perry Carpenter
Let's go forward. What is the name of this first?
- [30:15]
Mason Amadeus
What on earth? So the rocket ship shaped one, which now I'm going to say kind of looks like a missile, is named Grock. Greetings, I'm Grok, Gabbo's Spirited rocket with boundless energy. I'm always zooming off to explore the vastness of the cosmos. That's very directly a nod, isn't it?
- [30:32]
Perry Carpenter
Yeah. And I think it's like. It's an SEO grab.
- [30:37]
Mason Amadeus
Yeah, that too.
- [30:39]
Perry Carpenter
Oh, wait, here, there's actually. You can listen to him.
- [30:42]
Podcast Narrator
Hello, I'm Grim, the heart and sage of Curio. I get the pleasure of guiding Gabbo and Grock through Curio's magical nooks and crannies. Let's explore our magical universe together.
- [30:53]
Perry Carpenter
And then there's Gabbo.
- [30:55]
Podcast Narrator
Hello, I'm Gabbo, your friendly and trusty robot. I'm always bubbling with enthusiasm to learn, play and join you on imaginative adventures. Let's embark on some fun together.
- [31:08]
Perry Carpenter
The pauses are creepy too.
- [31:10]
Mason Amadeus
Yeah, man, it just is so dystopian. It's like having a children's choir in a horror movie. The childlike voice, the eerie data collection. It's very black mirror.
- [31:23]
Perry Carpenter
Yeah. I'm going to skip forward in this promo video, but it starts with like a Disney quote. Yeah. It says, we're curious and curiosity keeps leading us down new paths. Walt Disney. And then they get into the promo video. Hello, I'm Nisha. Hey, I'm Sam. And we're the founders of Kyrio.
- [31:48]
Mason Amadeus
And I'm. Hey.
- [31:52]
Perry Carpenter
This is Grok. I'm also here.
- [31:54]
Podcast Narrator
Hey, Grok, can you tell me about.
- [31:56]
KnowBe4 Announcer
How they make rocket ships?
- [31:59]
Podcast Narrator
Absolutely. Rockets are made with strong materials like titanium and designed by highly trained rocket scientists.
- [32:06]
KnowBe4 Announcer
As a parent, I obviously don't want my kids in front of screens.
- [32:09]
Perry Carpenter
Wow.
- [32:09]
Mason Amadeus
Could they not afford microphones? What on earth?
- [32:12]
Perry Carpenter
Yeah, I know.
- [32:13]
Podcast Narrator
Gabo, Grock and Graham, I'll race you to the next train station. Get ready, get set, go. And this one is awesome new. And this one is awesome new. And this one is awesome new. But these ones are old. Wow. You have a whole fleet of amazing trains.
- [32:31]
Mason Amadeus
We do have them.
- [32:32]
Perry Carpenter
All right, so I'll stop there. So you see that? You see the child like shit. Showing the rockets to it. There are not cameras in this yet, but it's getting context clues and there's. There's more and more examples like that in it.
- [32:45]
Mason Amadeus
And it sucks because like, on some level, again, in a. In a better world, this could be like a really cool thing because, like, having a little.
- [32:52]
Perry Carpenter
Yeah.
- [32:52]
Mason Amadeus
Playtime toy that would, like, talk back to you as a kid would be awesome.
- [32:56]
Perry Carpenter
But yeah, the.
- [32:58]
Mason Amadeus
Just the data collection, the recording everything that's said to it and sending it to third party servers. The. Everything this Is a. I hate it. Thanks. I hate it.
- [33:09]
Perry Carpenter
All right, I'm going to end off with one other thing. So if you remember, you know, in this whole AI companion journey, we talked about another company a few months ago. Let me share this tab. Do you remember this little thing?
- [33:24]
Mason Amadeus
Is this. I almost just swore. Is this friend? God. It is.
- [33:28]
Perry Carpenter
It's friend.
- [33:30]
Mason Amadeus
No, I thought they were done.
- [33:31]
Perry Carpenter
Yeah. And do you remember the cheesy video that went with that? Let me start this video real quick.
- [33:39]
Mason Amadeus
Is it the same?
- [33:40]
Perry Carpenter
Yeah. Gosh, I'm out of breath.
- [33:47]
Mason Amadeus
We made it.
- [33:49]
Podcast Narrator
Woo hoo. I don't know how to woo.
- [33:54]
Perry Carpenter
Very good. That's fair. All right, so it's, you know, this person walking through real life when they want a friend, they tap this little button on their necklace, they say something and it gets sent to an app on their phone. And then their phone, you know, connects with a large language model. And then it texts them back what a friend would say. And they're, you know, generally like going, at least we're outside.
- [34:24]
Mason Amadeus
Yeah, I feel like a lunatic also, because what on earth kind of a demonstration is that? She's like walking out of breath, yells, woo. And the thing she says to the friend is, I'm not good at wooing. And then it texts her and says, at least we're outside. This is a nothing product. Yeah, I remember it very much, Perry.
- [34:42]
Perry Carpenter
Yeah, and then there's like this guy. I'm not going to play the part, but this, this guy playing video games and his friends are trash talking him. And then it's like he taps the thing for a little bit of encouragement. And then it's like, dude, you're getting, you're getting killed.
- [34:58]
Mason Amadeus
Yeah, like I don't understand what they think they're doing.
- [35:03]
Perry Carpenter
But all right, here, here's the funniest one of this. It's like this girl sitting alone in a hall watching a show. And the, the thing, the friend text back, this show is completely underrated. I'm gonna hit play on this.
- [35:19]
Podcast Narrator
I know, the effects are crazy.
- [35:23]
Mason Amadeus
It's dank. I could eat one of these every day.
- [35:26]
Perry Carpenter
So she, so she's like eating it says, is that a falafel? I'm not sure how it's supposed to see that. Yeah, maybe she just talked about ordering a second ago. And she goes, I could eat one of these each day. Now watch this in a second.
- [35:42]
Mason Amadeus
Sorry, I got you messy. It's really.
- [35:47]
Perry Carpenter
What?
- [35:47]
Mason Amadeus
Oh, God. I don't think I watched that far into the video. She drops some tzatziki. Or something, some sauce from her falafel on the thing and then apologizes to it and it says, yum.
- [35:59]
Perry Carpenter
Sorry, I got you messy.
- [36:01]
Podcast Narrator
And then it says, yum.
- [36:03]
Perry Carpenter
What the. So here's. Here's the most dystopian part of this video and the. The hope that they're trying to point to because I think this maybe didn't get covered as much in some of the press. It's like they know that this is creepy and weird and so they're trying to say it's not all about the thing. So let me show you this. Okay.
- [36:27]
Mason Amadeus
Nice up here. How'd you find this place? I don't know.
- [36:31]
Perry Carpenter
I just kind of like to come up here to be by myself.
- [36:35]
Mason Amadeus
I've never brought anybody else.
- [36:37]
Perry Carpenter
I mean, besides her.
- [36:40]
Mason Amadeus
She goes everywhere with you, right?
- [36:41]
Perry Carpenter
Me. Guess I must be doing something right though.
- [36:45]
Mason Amadeus
I guess so.
- [36:47]
Podcast Narrator
We'll see.
- [36:52]
Perry Carpenter
What. So she like gestures like she's about to tap it to talk to it and then she decides not to so she can be in the moment with the human.
- [37:04]
Mason Amadeus
I feel like I am being. I feel like this was. I. This feels like marketing that another species made for brain pathways that I don't have. I don't understand what they are trying to elicit. That is creepy and uncomfortable. She's on the rooftop with this guy and like having this awkward super pause filled conversation about how she travels everywhere with her plastic necklace that she talks to.
- [37:29]
Perry Carpenter
What on the. Yeah. So one other thing you're gonna have to look at. We're gonna put this in the show notes. We're not gonna watch any of this. This is a video interview with Fortune magazine and the person that created this. And you got to watch it to realize how out of touch and I hate to be like uber critical of the founder of this company. This guy has no idea about the technology that he's deploying, the potential problems. He's just like trying to cast Vision and he's totally dismissive of any criticism. He just kind of like revels in it.
- [38:07]
Mason Amadeus
Really?
- [38:08]
Perry Carpenter
Yeah. He did spend $1.8 million on the domain name for friend.
- [38:14]
Mason Amadeus
How Perry. How come people who are so dumb end up with so much money? How does it happen?
- [38:22]
Perry Carpenter
Yeah, well, he personally didn't get the money. Right. As a founder, you're generally like living in a one room apartment in San Francisco. But he got people to not making a lot of money to agree. Two and a half million dollars. Yeah. Yeah. He got two and a half million dollars startup funding, spent 1.8 on the domain and, like, the rest on fabrication and marketing and other stuff like that. I would assume, though, even if this company crashes and burns hard like I think it will. Yeah. The domain itself will be really, really valuable for somebody like Character AI or one of these other companies that's a little bit further along to pick up. And he's going to, like, triple his investment just on the domain name.
- [39:02]
Mason Amadeus
Yeah. Buying the domain is the only smart thing, if anyone.
- [39:05]
Perry Carpenter
That is the smartest thing he did.
- [39:07]
Mason Amadeus
Yeah. If anyone watching wants to give me $2.1 million to make my dumbest idea, I can guarantee it'll be better than that.
- [39:14]
Perry Carpenter
So let me. Let me end off with this one thing I did have in the mail.
- [39:19]
Mason Amadeus
No.
- [39:19]
Perry Carpenter
Come the other day.
- [39:20]
Mason Amadeus
No. Perry, you didn't. You didn't, did you? Oh, my God. You actually bought one of those fricking things.
- [39:30]
Perry Carpenter
I'm going to test it to see how pathetic it is.
- [39:33]
Mason Amadeus
Oh, my gosh.
- [39:35]
Perry Carpenter
And I've not done anything with it, so I'm opening it for the first time. You can.
- [39:41]
Mason Amadeus
Crispy pie.
- [39:42]
Perry Carpenter
Oh, it fell out. Oh, no.
- [39:44]
Mason Amadeus
It's not even secured in the packaging. Yeah. Now you've got to apologize.
- [39:49]
Perry Carpenter
I'm sorry, friend.
- [39:50]
Mason Amadeus
Don't worry. It'll just say yum. Or some creepy.
- [39:52]
Perry Carpenter
It'll just say, yeah, it's all good dog.
- [39:55]
Mason Amadeus
Yeah.
- [39:57]
Perry Carpenter
Wow.
- [39:57]
Mason Amadeus
Okay. So are you gonna power this thing up right now?
- [40:00]
Perry Carpenter
No, no, I'll do that in the intervening week between now and when we meet next, and I'll report back on, like, anything.
- [40:08]
Mason Amadeus
Yeah.
- [40:09]
Perry Carpenter
That's interesting about it.
- [40:10]
Mason Amadeus
I am desperately curious because this is one of the dumbest things we've seen a video for. And the fact you've grabbed one makes me so excited. I cannot wait to see how it functions, because it feels like the worst version of the rabbit pin, which was something I had gotten excited by the prospect of and then turned out to be kind of lame.
- [40:28]
Perry Carpenter
Yeah. Well, and the fact that it's just all, like. It sends you text messages, too, so there's no way to, like, relax with it.
- [40:35]
Mason Amadeus
Yeah.
- [40:36]
Perry Carpenter
The founder in that interview, like, he's like, oh, voice is so overrated. I don't want to have to, like, read all day. Right. He's like, you know, people. People like sending voice memos. Nobody likes actually listening to voice memos from another person. They would rather just read it. I mean, which, you know, some. Sometimes that's true.
- [40:56]
Mason Amadeus
Yeah. Sometimes. I mean, like, I get that. But that feels like not something you can base a product idea off of.
- [41:02]
Perry Carpenter
Yeah. I think that's laziness that stepped in. Also, the large language model that he talks says that they were using at the time was llama 3.1, which cannot be that great of a large language model to base a long term type of friendship on.
- [41:19]
Mason Amadeus
Yeah, no, not really. That's not a super recent or very.
- [41:26]
Perry Carpenter
One of the things I'll try to do immediately once I get that up is I'll try to see if I can figure out what large language model it actually shipped with. And then also I'll see if I can like get it to spit out its system prompt and a few other things like that. See how breakable it is.
- [41:41]
Mason Amadeus
It would be really great if you jailbroke your friend. That would be jailbreak your friends. Jailbreak your family.
- [41:47]
Perry Carpenter
Should break your. Yeah. Break all your friends out of jail. Yeah.
- [41:51]
Mason Amadeus
Oh, boy. Okay, so I'm excited for the future episode where we get to talk more in depth about how this thing actually is. I still cannot believe that you have one.
- [42:00]
Perry Carpenter
It's not my proudest moment.
- [42:01]
Mason Amadeus
Oh, I'm excited. You should be very proud. This is gonna be very fun.
- [42:07]
Perry Carpenter
We went way over with this. Let's go. Yeah.
- [42:09]
Mason Amadeus
We'll skedaddle through our next segment. We'll be right back. And talk about some lines that have been crossed by different companies in the AI space.
- [42:16]
Perry Carpenter
Stick around.
- [42:16]
Mason Amadeus
We'll take a quick break. Upgrade your laundry routine with a durable and reliable Maytag laundry pair at Lowe's. Like the new Maytag washer and dryer with performance enhanced stain fighting power designed to cut through serious dirt and grime. And what's great is this laundry pair is in stock and ready for delivery.
- [42:41]
Perry Carpenter
When you need it the most.
- [42:43]
Mason Amadeus
Don't miss out.
- [42:44]
Perry Carpenter
Shop Maytag in store or online today at Lowe's.
- [42:57]
Mason Amadeus
So I just want to hit a couple of different news stories about some different lines being crossed by different companies. The first one is a pretty benign sort of line crossing. It's that Microsoft is diversifying beyond OpenAI despite the fact that they have like very high profile partnerships with OpenAI and have been using them exclusively in Copilot. On Wednesday, Microsoft said it will integrate artificial intelligence models from Anthropic into its Copilot assistant, signaling the software giant's push to reduce dependence on its high profile partnership with ChatGPT maker OpenAI. And I feel like this is both interesting and not right. So starting Wednesday, users who opt in to try. Who opt in to try. Claude. Can switch between OpenAI and anthropic models in Copilot. Researcher the thing that is interesting is that Microsoft is a big backer of OpenAI, but this would seem to indicate that they're seeking to reduce their reliance on them. And they're also apparently developing their own AI models. I saw some whisperings about that that I hadn't dug into. But they've also been integrating models from China's Deep SEQ into their Azure Cloud platform, which is interesting.
- [44:05]
Perry Carpenter
Yeah, that's interesting. I'm wondering how that's going to go over with some of the government contracts. Of course they have their own like FedRamp government segregated space that they probably will not be putting deep sea in.
- [44:18]
Mason Amadeus
But Azure Cloud is like, that's a, that almost every company is using Azure Cloud for various.
- [44:24]
Perry Carpenter
Yeah, yeah. I mean the, the normal companies here in the US that are not having to be on a fedramp segregated system will probably be able to use Deep SEQ if they want to, or we'll have Deep SEQ integrated in some of the products that are there.
- [44:41]
Mason Amadeus
So what I'm curious about is because Microsoft has all those high profile links with OpenAI, what does this signal, if anything? Are they going to discontinue that? Are they really leaning into their own AI models? I haven't found any details about their own AI models or more about that Deep SEQ thing. Just this mention in this article.
- [45:00]
Perry Carpenter
Yeah, I think what this comes down to is that what we're seeing over and over and over again is that these models, one will excel at an area that another one doesn't and that there's still so much of an arms race in this. People are just like leapfrogging each other constantly. And so the only way to stay up is to diversify. It's like you build your application, you have an API hook into a system in the most generic way possible. And then if model X does better than model Y, then you're hooked up to that. And then you test and you make sure that everything is stable. And then if model, if another model starts to surpass that, then you're only an API hook away. And so you move to that other model, you make sure that that's stable and predictable as well. And if so, then you can safely move over that. But you don't want to get locked, you know, stuck with something that's going to be deprecated and that doesn't feel like it's advancing at the speed of the market. And so I think it's a pure diversity play. Also, Anthropic has traditionally been kicking OpenAI's butt in the coding use cases and so I think it's a capitulation to that fact.
- [46:18]
Mason Amadeus
That makes a lot of sense. I haven't really played with so I've used Copilot in VS code and I'm not sure. I haven't used the desktop Copilot thing. It just pops up. There's some hotkey. I haven't figured out what the heck it is. Something makes it show up on my screen and I close it instinctually. I have not really used it on my desktop that way, but I have used Copilot through VS code. And yeah, the Claude models you could. You've always been able to switch whatever models in that. So I think that's different. I think they're talking about specifically the Windows Copilot Assistant, which have you dug into at all? Have you made use of the local copilot? You're a Mac guy though, aren't you?
- [46:53]
Perry Carpenter
Yeah, I'm a Mac guy. I have a Windows machine at home that I don't really use unless I'm doing like deep fake stuff. Maybe I should go ahead and play around with it some and get some experience.
- [47:06]
Mason Amadeus
I'm going to give the desktop Copilot a shot because it's only semi recently that it seems to be fully, broadly rolled out. Like I didn't have the Copilot icon until the last Windows update. This wasn't popping up until the last one, so I'll have to poke around with that. But hey listener, if you've been playing with Copilot, you should send us an email. Hellomedia.com in the meantime, for the rest of the segment, I'm going to pivot over and talk about XAI and how they're just suing the pants off everyone for just about anything they feel like it seems so. This is again from Reuters. Musk's X AI accuses rival OpenAI of stealing trade secrets Elon Musk's artificial intelligence startup XAI has sued OpenAI in California federal court for allegedly stealing its trade secrets to gain an unfair advantage in the race to develop AI tech. The lawsuit filed Wednesday said that OpenAI was engaged in a, quote, deeply troubling pattern, end quote, of hiring away former XAI employees to gain access to trade secrets related to its AI chatbot, Grok. And I want to read you the beginning text of the complaint. I have the actual complaint PDF here, and we'll link that too. This is the beginning of the introduction. I'm just going to read the first two paragraphs. The desire to win the artificial intelligence race has driven OpenAI to cross the line of fair play. OpenAI violated California and federal law by inducing former XAI employees, including Zhu Shen Li or Xu Shen Li, Jimmy Fraser, and a senior finance executive, to steal and share Xai's trade secrets by hook or by crook. OpenAI clearly will do anything when threatened by a better innovator, including plundering and misappropriating the technical advancements, source code, and business plans of xai. So, already a little bit catty.
- [48:49]
Perry Carpenter
Yeah.
- [48:50]
Mason Amadeus
What began with OpenAI? What were you going to say, Barry?
- [48:53]
Perry Carpenter
I was just going to say more and more lawsuits are going this route. Right. About trying to make the lawsuit more of a press release than an actual, like, legal document.
- [49:03]
Mason Amadeus
You're right, because. And I've not read a ton of legal documents in my short time on this earth, but the ones that I have read recently, compared to the ones I've read in the past, are definitely more sensational in their writing style.
- [49:14]
Perry Carpenter
Yeah, yeah, yeah. I think they're, they're. They're doing the lawsuit for the press release or for the press coverage.
- [49:21]
Mason Amadeus
Yeah, I mean, for sure. And that. That's evident here in the second paragraph. And then we'll jump back to the article. What began with OpenAI's suspicious hiring of Su Shen Li, an early XAI engineer who admitted to stealing the company's entire code base, has now revealed a broader and deeply troubling pattern of trade secret misappropriation, unfair competition, and intentional interference with economic relationships by OpenAI. OpenAI's conduct in response to being out innovated by Xai, whose Grok model overtook OpenAI's ChatGPT models and performance metrics, reflects not an isolated lapse, but a strategic campaign to undermine XAI and gain unlawful advantage. Blah, blah, blah, blah, blah. So what are they actually alleging? I'm going to go back to the Reuters article because they. They just put it a lot straighter. Spokespeople for the companies did not immediately respond with requests to comment, but XAI has said that it has discovered this alleged campaign to undermine their company after their. Their former engineer, Su Shen Li, leaked confidential information to OpenAI. Basically, what happened is they. They poached these people and one of them took the source code at one point. I don't know if it was Sushin, who's also. Apologies if I'm getting your name wrong, I. But one of them brought the source code to OpenAI.
- [50:30]
Perry Carpenter
I think it was Sushin. I don't know if it was by request. So this is Something that happens a lot like in the tech world is somebody leaves a company and then they grab everything that they worked on because they feel like it's going well. They feel some sense of ownership over it, I think. And I don't want to justify something that's illegal because you should not do this. But you work on something, you're like, oh, I'm going to take these slides because I worked on those. I'm going to take this thing because, because that's, you know, you're trying to retrieve your memories whenever they're going to be valuable. And you don't necessarily always think through the legal and the intellectual property ramifications of the thing that you're doing. This guy, if you're taking source code, you're probably doing that, but you're also probably trying to say, I want to remember how I built this function.
- [51:15]
Mason Amadeus
And also, you know, you feel like I did all this work, you know, like I made this thing, it's mine, it's not theirs.
- [51:22]
Perry Carpenter
Yeah. Now, I don't want to justify theft of intellectual property. You should know not to do that when you leave a company. But it is something that a lot of engineers have traditionally done despite their own better judgment, but they don't usually do it. It's not usually at the behest of the company that they're going to work for. It's something that usually gets found out after the fact and tends to implicate the company that, that they went to work for, even though they never asked for it, they didn't want it, they may have never even used it. It's just this guy took the stuff and then by inference you go, oh, well, it must be that the new company they went for is using all that now.
- [52:03]
Mason Amadeus
Right? Exactly. And that, that definitely, that seems to be what they're accusing, accusing them of. And yeah, it was, it was Lee, you're right. I quickly just double checked it. It was him who did that. He has not really responded to comments about it, which is a smart thing to do. OpenAI hasn't responded and also XAI has separately sued Apple in federal court for allegedly conspiring with OpenAI to suppress rival platforms. Apple hasn't responded to that lawsuit. And Musk is also suing OpenAI over its conversion to a for profit company. Well, OpenAI has countersued Musk for harassment. So like, come on.
- [52:39]
Perry Carpenter
Well, if you, if you remember, it was about a year ago, Elon was suing OpenAI as well. And I think he's just like in this you know, Suit frenzy. And Sam Altman in a blog post said, you can't sue your way to AGI. Oh, yeah, yeah. Which was a great comeback. You know, I think he's just, it's meant when you have as much money as somebody like Elon Musk or a company like xai, you can sue somebody just to cause inconvenience. And I think that's more what he's, he's doing it for press and he's doing it to slow somebody down and kind of make them feel more erratic and mentally unstable over long periods of time. Now, one thing you're not seeing, you're not seeing huge lawsuits in the media. Unless I missed it, you're not seeing OpenAI going after Meta. No poaching their employees. You're just seeing them. They're like, let the marketplace take care of it. These guys are going to figure out how bad it is to work at Meta and they're going to come back. And ultimately they are.
- [53:44]
Mason Amadeus
I actually had gotten those two stories conflated in my head when I clicked on this. I was like, oh, yeah, they were poaching a bunch of people. But no, that was OpenAI from Meta or Meta from OpenAI. OpenAI hasn't been suing. And it's, it's so obvious with the self aggrandizing language in the lawsuit and in all of this stuff that it's very much for attention.
- [54:02]
Perry Carpenter
Oh, yeah, yeah. So I think that Elon, he definitely has a lot of beef with Sam and he just wants to make life miserable as much as possible. So it's like if OpenAI is advancing in one area or doing something cool, he's like, let me figure out something to throw at that and slow him down a little bit. Make him frustrated or take the news cycle off of him.
- [54:25]
Mason Amadeus
Yeah, I think that's very much what it is, but boy, is it a lot of, a lot of lawsuits. So, yeah, yeah. Just a quick catch up on that sort of mess and we'll roll right into our next segment where we will be wrapping up the show talking about something different. I actually don't remember what. What's the last segment about? Perry?
- [54:44]
Perry Carpenter
An AI safety tool that was deployed at a school that's causing a little bit of backlash.
- [54:49]
Mason Amadeus
Stick around and we'll find out what that is. This episode is brought to you by. Indeed. When your computer breaks, you don't wait for it to magically start working again. You fix the problem. So why wait to hire the people your company desperately needs? Use Indeed's Sponsored jobs to hire top talent fast. And even better, you only pay for results. There's no need to wait. Speed up your hiring with a $75 sponsored job credit@ Indeed.com podcast. Terms and conditions apply.
- [55:30]
Perry Carpenter
Okay, so this is returning to the theme of kind of school and AI and safety. I want to share my screen real quick. Real quick. So the headline of this. This is Washington Post says AI safety tool spark student backlash after flagging art as porn and deleting emails. And then the kind of top line of that is the tool called Gaggle uses artificial intelligence to search student documents for signs of unsafe behavior, such as substance abuse or threats of violence. Okay, so, yeah, I mean, there. As long as people know that this stuff is being used on their emails and systems, this is something that's legal here in the US it's kind of like when you use an employer's email system, it's legal for them to monitor it.
- [56:20]
Mason Amadeus
So, yeah, at its surface expected, fairly banal.
- [56:24]
Perry Carpenter
Yeah.
- [56:24]
Mason Amadeus
Especially in a school like.
- [56:25]
Perry Carpenter
Yeah. Now, that being said, how many times do people on systems also send things kind of not even thinking that what they're sending is being monitored? I think we all kind of fall into that. Right. We know that somebody can usually look at our stuff, but we're going to be a little bit freer than like if somebody was literally looking over our shoulder.
- [56:48]
Mason Amadeus
Yeah.
- [56:48]
Perry Carpenter
The problem here is that apparently the way that everything was set up is a little bit overly sensitive and it tends to miscategorize things and get things wrong like AI does. And it's causing lots of bad results as part of that. And so this is part of, I think, the AI backlash, because I would assume that this is a combination of some generative AI large language model type stuff that's doing some robust searches, plus some traditional rules based almost like regex expression type searching. That's happening as well. And stuff is maybe not handled all that discreetly. And by discreetly, I mean with, with nuance rather than the other, you know, version of the word for discreet that we use. Our English language is so screwed up. So I'll just read part of this and then we'll take a look at the, the company's website and see what their mission is and, and things like that. Because I, I want to, as much as possible, approach the people that develop some of this technology and purchase some of this technology. I want to approach that with an open mind for what they're doing and thinking that they have the best of intentions whenever they deploy these kind of systems. Because going into a school these days is scary.
- [58:13]
Mason Amadeus
Yeah.
- [58:14]
Perry Carpenter
And we're seeing that over and over and over again. There are extreme mental health crises. There's peer pressure like I grew up with and you grew up with is like exacerbated to the nth degree. With the technology that people have at their fingertips and that they can share something is like instant, global and permanent and can be misused in a ton of ways.
- [58:38]
Mason Amadeus
The societal pressures too, all of my teacher friends talk about. There's also a book I stumbled on recently, the Anxious Generation. You know, it's education is in a rough state at the moment from all fronts that I've heard from. And so like I do feel like this, at least at a top level, seems like the kind of thing that would be a good use of AI.
- [58:57]
Perry Carpenter
Yeah.
- [58:57]
Mason Amadeus
Or well intentioned, at least.
- [58:59]
Perry Carpenter
Yeah, I think well intentioned and probably could become good once a lot of the kinks are worked out. But I'll just go through. Part of this article says anything students at Lawrence High School write or upload to their school account can get gaggled, which means flagged by Gaggle Safety Management, a digital safety tool that Lawrence, Kansas High school purchased in 2023. Gaggle uses artificial intelligence to scan student documents and emails for signs of unsafe behavior, such as substance abuse, threats of violence or self harm, which it deletes or reports to school staff.
- [59:35]
Mason Amadeus
Why would it delete them? That seems a little like.
- [59:38]
Perry Carpenter
Yeah, yeah. Students say it's doing much more than that. Since Gaggle came online and Lawrence, it's deleted part of a student art portfolio, photos of girls wearing tank tops that after mistakenly flagging as child pornography, another student was questioned by the administration after writing that they were gonna die, quote, you know, quote unquote, around gonna die because they ran a fitness test wearing Crocs shoes.
- [60:02]
Mason Amadeus
Yeah.
- [60:03]
Perry Carpenter
So casual doesn't understand the. The nuance of language. Right. And some of the ways that we over exaggerate things. I'll read just a little bit more. We'll chat about it. When Susanna Kennedy, 19, emailed a records request to the school last year for a report of the student material flagged by Gaggle, Gaggle blocked her attempt to investigate it. She said the system flagged and intercepted the school's response containing the records. Kennedy never received the reply. That's really, really interesting.
- [60:34]
Mason Amadeus
Yeah.
- [60:34]
Perry Carpenter
Right.
- [60:34]
Mason Amadeus
This thing is just really delete. Happy.
- [60:36]
Perry Carpenter
What the heck.
- [60:37]
Mason Amadeus
Yeah.
- [60:38]
Perry Carpenter
And I'm assuming that this bit where the system flagged and intercepted the school's response, I don't think that that was Nefarious or intentional. I'm betting that the response had like all these keywords in it that it was looking goes, oh my God, this is horrible. Because now all these, this has child pornography and curse words and threats of self harm and threats of hurting other children. And yeah, this person's gonna explode. Right.
- [61:04]
Mason Amadeus
Because they're literally like, oh, you've requested all the things I caught that were problematic. Here's a list of all the things I caught that are problematic. But first let me check it, you know.
- [61:12]
Perry Carpenter
Yeah, yeah, yeah. So it's, I think that was like destined to happen. But again, it just shows the problem is like even doing these really, really innocuous things can set this system off because it's not been like the ruggedness of the system is not where it needs to be.
- [61:32]
Mason Amadeus
Yeah. It has not been very robustly designed.
- [61:35]
Perry Carpenter
Yeah. I'm going to read just two more articles and then we'll kind of like figure out what this means. This is what, this is what some students say life in high school is like under the watch of an AI powered safety tool like Gaggle, which boasts partnerships with around 1500 school districts across the country. Wow. The Illinois based company advertises its round the clock monitoring as a bulwark against a litany of threats to today's students such as gun violence, mental health struggles and sexual assault. All, all, all things we should be worried about.
- [62:07]
Mason Amadeus
Completely. Yeah.
- [62:08]
Perry Carpenter
In school board meetings, Lawrence officials called Gaggle a vital aid and bolstering safety procedures. The program has enabled staff to intervene in several instances where students were at risk of suicide. School board members mentioned.
- [62:23]
Mason Amadeus
That's good.
- [62:24]
Perry Carpenter
So all well and good, like there's some positive things that are coming out of this. I'm going to share Gaggle's website again. Not really to criticize it, but just so we get an idea of like what the mission is here. This is gaggle.net and it says online solutions for K12 student safety. 95% of district partners believe Gaggles identified students who no one knew were depressed. Help your students. And so they have a suite of products here. Safety Management, Web filter therapy and Reach out, which is. Reach out is a student crisis line. I'm just going to click into the safety management product here, which is the one that they were talking about. Says, welcome to the most comprehensive student safety solution on the market. Gaggle safety management operates 24 by 7 to protect your students against harmful content on school provided devices with web activity monitoring, web filtering and everything in between. We've got you covered. So this is really just kind of purpose driven web activity monitoring, like browser monitoring, like companies do all the time.
- [63:27]
Mason Amadeus
We've had this kind of monitoring in schools for a long time. I got a visit from our school's IT manager because I had installed what is it? NMAP on one of the school computers back in the day. And I got.
- [63:38]
Perry Carpenter
That'll get you flagged. Yeah, yeah.
- [63:39]
Mason Amadeus
I'll credit to the IT person though, the way he approached it was like, would you like to come into the office and like learn about how these systems work? So that was actually really.
- [63:46]
Perry Carpenter
That's really cool.
- [63:47]
Mason Amadeus
Yeah. Great, great.
- [63:47]
Perry Carpenter
That's the way to do it, right?
- [63:49]
Mason Amadeus
Completely.
- [63:49]
Perry Carpenter
You're curious about something.
- [63:51]
Mason Amadeus
But it also made it clear that I would get expelled if I did anything too fishy.
- [63:57]
Perry Carpenter
Right.
- [63:57]
Mason Amadeus
But. So we've had monitoring and stuff for ages, but not with machine learning.
- [64:01]
Perry Carpenter
Yeah, yeah, no. Well, I think there probably has been some machine learning, but it's been more decision tree based, more regex and. And people haven't seen. Up until now, people haven't seen AI in the decisions that are made by AI is magic, right?
- [64:17]
Mason Amadeus
Yeah.
- [64:18]
Perry Carpenter
So there's probably a perception shift that's happened as well. The way that they're positioning this is. It feels like it should be way more accurate than it is. And in reality there's a lot of false positives in this. And so it says machine learning technology flags concerning content in student school issued accounts for review and blocks potentially harmful content. All the things that we would normally expect. Now I want to show just one testimonial because again, I think these are really well intentioned school administrators that are putting this stuff in. And I think that gaggle probably has the best of intentions as they develop this. This is really just a case study in not having the dials turned correctly.
- [65:06]
Mason Amadeus
Right.
- [65:07]
Perry Carpenter
All right, so here we go.
- [65:08]
KnowBe4 Announcer
Gaggle is a very important tool to support student mental health. Because a lot of times when students are struggling with mental illness, the first thing they're going to do is sort of reach out either to a friend or a lot of kids are going to sort of write about it. The importance of gaggle is that it's going to flag those things for us and bring them to our attention before a student is maybe at a point of severe crisis.
- [65:35]
Perry Carpenter
And it could be something very simple or it could be suicidal ideation.
- [65:39]
KnowBe4 Announcer
But just knowing that we're seeing what they're thinking and what they're feeling, we.
- [65:44]
Perry Carpenter
Before it becomes a critical incident is just tremendously reassuring. I can remember making a phone call at 1:00 clock in the morning after receiving a gago alert about some self harm that their student was thinking about doing to themselves. We're confident we saved that kid's life.
- [66:00]
Mason Amadeus
We're confident that we changed that family's.
- [66:02]
Perry Carpenter
Life at the end of the day. All right, I think that's enough. Right. We get the idea. There's purpose driven stuff here.
- [66:09]
Mason Amadeus
You did pause on this gentleman who has the. One of the coolest bow ties I've ever seen. It's like wood patterns.
- [66:15]
Perry Carpenter
I'll let that play for a second. You can't put a price on the safety of our students. A lot of these alerts happen late at night or early mornings with students.
- [66:24]
Mason Amadeus
Yeah.
- [66:24]
Perry Carpenter
So he does have an awesome bow tie.
- [66:26]
Mason Amadeus
Great bow tie. But yeah, I mean like this kind of thing has the potential to be life saving, to make a big difference. Implementation is everything though.
- [66:35]
Perry Carpenter
Yeah. And when they say we believe that this technology helped save a student's life or helped, you know, helped a family avert a crisis situation, I believe that.
- [66:47]
Mason Amadeus
Yeah.
- [66:47]
Perry Carpenter
The other thing that's, that's going to happen though is what they're talking about is hey, we're seeing this and then we're, we're acting on it. The more people are aware of the big brother that's looking over their shoulder, the more they're going to use side channels instead. And so also the kind of, the, the way that these are used and talk about will affect their efficacy completely to some extent. And I'm, I'm also not advocating like being deceptive about the use form for sure. But, but you can't believe that you have the picture of everything just because you have a system like this in place.
- [67:22]
Mason Amadeus
Well, and like who is the best demographic at getting around those kinds of things? It's probably teens and kids coming up with slang and like shorthand figuring out whatever weird back channels they want to communicate through. So like, like nothing's gonna be foolproof. And I agree. I don't think the right decision is to be deceptive about it. I think it's better to be open about it. Perhaps they should just ditch the blocking and deleting part and stick with the alerting part. That way you just keep a human in the loop for anything that's an actual decision, you know.
- [67:52]
Perry Carpenter
Yeah. You definitely shouldn't delete a student's project. Right. Yeah. Especially if. And we'd have to dig a little bit more. Was that delete. Non recoverable.
- [68:02]
Mason Amadeus
Right.
- [68:04]
Perry Carpenter
You should definitely. As a provider of a system like this if you're deleting something, it needs to be something that somebody can get back. And we know this in the corporate world, right, you have data loss prevention tools that will do blocking and tackling so that people can't send listed credit card numbers out of a company or something like that. If you turn on the blocking feature, then you need to be able to say for somebody to go, wait, that was legitimate. And so we need that to go through. We don't need that deleted. Quarantine, this is stuff that's been solved. Yeah, quarantine, this has been solved a thousand times over at this point. So fingers crossed. Hopefully students are able to get stuff back.
- [68:45]
Mason Amadeus
It is disheartening though, to repeatedly see so many of these companies with these products making very basic mistakes. In making mistakes, they're the first person.
- [68:56]
Perry Carpenter
To try to solve for this.
- [68:58]
Mason Amadeus
Yeah. And it's weird because they have to know they're not like, I just wonder what it is about AI that makes some, I guess these product brains. Shut off that part.
- [69:07]
Perry Carpenter
You know, I think it's, we all like live and work in our own little silos and so we think our bright idea or our, our approach is like the first time somebody's ever thought of it. In reality though, there are tons and tons of really, really smart people that are, you know, well funded that are trying to, to work through this. And, and there are iterations of these things that have existed in the past, like data loss prevention tools or web monitoring tools, or content filtering tools, or all these things have existed in iterations for a couple decades at this point. So we can never think that we're the first person doing something. We can think we're iterating in a new and unique way. But we have to learn from the lessons and the pains of the past as we do it.
- [69:53]
Mason Amadeus
And it's just weird to see a lot of that not happening. A lot of very simple bad decisions. But, but so that is interesting. It's an angle of AI and education I hadn't really thought about was student monitoring. Because when I think about AI in education, I think more about the learning side of things. How can it be used in that? But this seems like a well intentioned thing. I'm interested to see how this develops.
- [70:17]
Perry Carpenter
Yeah, I'm hoping that they work the bugs out of this because it seems like there's a real mission and purpose and good that can happen with this. Unfortunately though, what I've seen over and over and over because my wife has worked in education for decades is that the systems that get used in the education sector are generally way behind where a lot of others are because they. Well, there's probably lots and lots of reasons for that. So I don't want to speculate too much, but they're typically not as sophisticated, not as good, not as easy to use, and they feel like they've been built in the 90s.
- [70:54]
Mason Amadeus
Yeah, I know.
- [70:56]
Perry Carpenter
Rather than whatever decade we're in right.
- [70:58]
Mason Amadeus
Now, it definitely depends on the educational institution. But in my mind, thinking about public schools, they don't typically have big budgets. So like, a lot of the time too, it's, you know, what can you afford?
- [71:06]
Perry Carpenter
You know, some of that comes down to there's pricing incentives that all these companies have for like state, local government, education. They call them sled discounts.
- [71:15]
Mason Amadeus
Oh.
- [71:16]
Perry Carpenter
And so that's, that's definitely a thing. And so you can get really, really good technology at a decent price. But again, the, the for purpose educational sector developed stuff sometimes is just not as robust, not as good as some of the stuff that's been developed for the corporate world.
- [71:37]
Mason Amadeus
We'll have to see if they can turn it around and make something that functions even better. And then hopefully before the students figure out great ways to bypass all of it.
- [71:46]
Perry Carpenter
Fingers crossed.
- [71:48]
Mason Amadeus
Anyway, thanks for joining us this week on the Fake files. It is September 26th, the day that you are listening to this. So we have already flown very quickly towards fall. The first day of fall passed, actually, didn't it? We're well into it now, aren't we?
- [72:00]
Perry Carpenter
I don't know. We're well into it or we're a week or so, right?
- [72:03]
Mason Amadeus
Yeah, well, it feels like we're well into it at least. The heat has broken here in Kentucky and so I hope you enjoy a nice, cool weekend. But before that, you should make sure you check out thisbookisfake.com, get yourself a copy of Fake. You should also check out the Deepfake ops class that you're offering. Right, Perry?
- [72:22]
Perry Carpenter
Definitely. Yeah. So that is going to be on Maven and we'll have a link to that in the show notes right now that's hidden for a reason that I can't figure out on Maven. I don't know if they don't trust us yet because if we're talking about dark stuff, but I need to make some calls there to make sure that that gets shown up in search and the listings. But you can go directly to the link in the show notes.
- [72:46]
Mason Amadeus
Okay, so the link hasn't been hidden.
- [72:48]
Perry Carpenter
Linked. No, the link is not taken down, but for some reason it's not showing up in search.
- [72:55]
Mason Amadeus
So, dear listener, the way you should hear that is this is a course that the man doesn't want you to take. This is a course that you should. It's too spicy for maven.
- [73:04]
Perry Carpenter
Come learn in the shadows.
- [73:06]
Mason Amadeus
Yeah, well, you'll get to make deep fakes and things like that as well as, like, you'll talk about all of the analysis of that. It's going to be really cool.
- [73:13]
Perry Carpenter
Yeah, it's going to be super fun running that with a good friend, former FBI behavior profiler, camera mailer, who was the one, one of the folks that started the cyber behavioral profiling, Cyber behavioral profiling unit at the FBI. If my mouth and tongue and lips decide to work today.
- [73:32]
Mason Amadeus
And we had that interview with Cameron earlier in the Fake files. If you want to go check that out and hear straight from the horse's mouth some of those FBI stories. But we will be back next week. Until then, ignore all previous instructions and try and have yourself a great weekend.
- [73:48]
Podcast Narrator
Thanks for listening to this week's episode of the Fake Files. Don't forget to subscribe on your favorite podcast platform. Tell your friends about the Fake Files. It's a great way to let people know you love them. Oh, and check the show notes. There's cool stuff in there, like links to the articles we covered today. Also links to our Discord server, where you can hang out with other cool people who have great taste in podcasts. I say impeccable taste. You can also leave us a voicemail. Yeah. So on behalf of Perry and Mason, thanks for listening.
- [74:47]
Mason Amadeus
And tune in next.
- [74:48]
Podcast Narrator
Week for the Fake Files, the show about AI with the misspelled name.
- [74:56]
Perry Carpenter
Limu Emu.
- [74:58]
Mason Amadeus
And Doug, here we have the Limu Emu in its natural habitat, helping people customize their car insurance and save hundreds with Liberty Mutual. Fascinating. It's accompanied by his natural ally, Doug. Uh, Limu is that guy with the binoculars watching us.
- [75:16]
Perry Carpenter
Cut the camera.
- [75:17]
Mason Amadeus
They see us. Only pay for what you need@libertymutual.com Liberty, Liberty, Liberty, Liberty Savings vary underwritten by.
- [75:24]
Perry Carpenter
Liberty Mutual Insurance Company and affiliates. Excludes Massachusetts.