AI Vulnerabilities and the Gentle Singularity: A Deep Dive with Project Synapse

Summary

Podcast Summary: Cybersecurity Today – "AI Vulnerabilities and the Gentle Singularity: A Deep Dive with Project Synapse"

Episode Details:

• Title: AI Vulnerabilities and the Gentle Singularity: A Deep Dive with Project Synapse
• Host: Jim Love
• Release Date: June 21, 2025
• Description: This episode explores the intricate vulnerabilities within AI systems, the concept of the gentle singularity as proposed by Sam Altman, and the urgent need to integrate robust cybersecurity measures into AI development to safeguard businesses in an increasingly digital landscape.

---

1. Introduction to Project Synapse and Episode Focus

Jim Love introduces the episode by highlighting Project Synapse, a recurring discussion platform where he, Marcel Gagne (an open-source and emerging AI expert), and John Pinard (head of operations and cybersecurity at a financial institution) delve into the latest trends in AI. Jim emphasizes the critical vulnerabilities emerging in AI systems that necessitate this deep dive into cybersecurity aspects.

Notable Quote:

Jim [00:00]: "There are some huge vulnerabilities in AI systems, most of which we are just beginning to think about."

---

2. Understanding the Gentle Singularity

The conversation kicks off with an exploration of Sam Altman's recent blog post titled "The Gentle Singularity." The panel dissect the concept, contrasting it with traditional notions of the singularity.

Key Points:

• Singularity Defined: Traditionally, the singularity refers to a point in technological growth where AI surpasses human intelligence uncontrollably, akin to the event horizon of a black hole beyond which nothing can escape.
• Gentle Singularity: Altman's perspective suggests a more controlled and gradual convergence between human and artificial intelligence, where AI surpasses human capabilities but remains aligned with human intentions.

Notable Quotes:

Marcel [02:32]: "Sam Altman has come up with what he calls a 'soft, gentle singularity.'"

Jim [04:54]: "We're heading toward this technological singularity."

---

3. AI's Intelligence vs. Human Intelligence

The panel delves into how AI models, particularly large language models (LLMs), exhibit forms of intelligence that mirror and sometimes surpass human cognitive abilities.

Key Points:

• Emotional Intelligence: AI models can detect and interpret human emotions with high accuracy, often outperforming humans in tasks like emotional recognition.
• Hallucinations in AI: Marcel compares AI's tendency to "hallucinate" — generating plausible but incorrect information — to human cognitive biases and the subconscious patterns that drive human creativity and errors.
• Daniel Kahneman’s Insights: Jim references "Thinking Fast and Slow" to draw parallels between AI's rapid, pattern-based responses (System 1 thinking) and human intuitive thought processes.

Notable Quotes:

Marcel [11:05]: "Large language models are even better than us at emotional intelligence recognition."

Jim [09:43]: "Our brains work all too much like we accuse AI of thinking."

---

4. Apple’s Critical Papers on AI Limitations

Jim brings attention to two recent papers published by Apple that critique the current state and limitations of AI models.

Key Points:

• Illusion of Thinking: One paper discusses how advanced reasoning models struggle with complex puzzles and generalizing reasoning, revealing significant accuracy issues.
• Model Context Protocol Vulnerabilities: The second paper highlights how AI models can be manipulated through prompt injections, where malicious inputs bypass safety protocols, leading to data leaks or unauthorized actions.
• Zero-Click Vulnerabilities: Specific vulnerabilities like Echo Leak exploit AI's tendency to accept and act on seemingly benign inputs without user interaction.

Notable Quotes:

Marcel [18:17]: "The paper you're talking about is called the Illusion of Thinking."

Jim [27:30]: "There's a lot of structural flaws that offer essentially zero protection."

---

5. Security as a Core Component in AI Development

The discussion underscores the necessity of integrating security measures directly into AI systems rather than treating them as secondary additions.

Key Points:

• Built-In Security: Just as traditional software development emphasizes security from the outset, AI systems require inherent security protocols to prevent vulnerabilities.
• Prompt Injection Risks: Techniques like embedding malicious prompts within images or hidden text exploit AI's scope violations, posing significant security threats.
• Agentic Frameworks: Implementing agentic frameworks allows AI to retrieve information externally, mitigating the risks associated with internal knowledge limitations and reducing the likelihood of hallucinations.

Notable Quotes:

Jim [46:04]: "Build it in now, save everybody a whole lot of time."

John [48:05]: "Incorporate security by running it in parallel with the development."

---

6. Business Culture and Ethical Responsibility

Jim, Marcel, and John critique the prevailing corporate culture that prioritizes shareholder value over safety and ethical considerations, drawing parallels to historical business failures.

Key Points:

• Shareholder Primacy: The legacy of CEOs like Jack Welch who emphasized shareholder value often leads to cost-cutting measures that compromise safety and integrity.
• Case Studies: Examples like the Ford Windstar crisis illustrate the catastrophic consequences of neglecting safety for financial gain.
• Philosophical Shift Needed: There’s a pressing need for businesses to adopt philosophies that prioritize long-term safety and societal well-being over immediate financial returns.

Notable Quotes:

Marcel [51:38]: "Companies make these calculations on the value of a human life or how much the company is going to suffer."

Jim [54:10]: "Jack Welch's focus on shareholders became gospel in business, leading to neglect of ethics and safety."

---

7. AI Security Challenges and Future Outlook

The panel discusses the escalating challenges in securing AI systems and the potential consequences if these issues remain unaddressed.

Key Points:

• Expanding Attack Surface: As AI integrates deeper into corporate infrastructures, the potential for large-scale cyberattacks increases exponentially.
• Human Element in Security: Emphasizing that human behavior remains the weakest link in cybersecurity, especially with AI facilitating more sophisticated social engineering attacks.
• CISO Evolution: The role of Chief Information Security Officers (CISOs) must evolve to include not just technical expertise but also strong interpersonal and psychological skills to address the nuanced challenges posed by AI.

Notable Quotes:

Jim [32:22]: "This could be a real stumbling point for corporations that start to say, wait a minute, our auditors are saying you can't touch this stuff."

Marcel [35:46]: "Security is the social engineering aspect of things, not just the technical engineering."

---

8. Conclusion: A Call to Action for Secure AI Integration

In the closing segment, the hosts emphasize the urgent need for the cybersecurity community to prioritize AI security, advocating for proactive measures and cultural shifts within organizations.

Key Points:

• Proactive Engineering: Encouraging businesses to implement security measures in tandem with AI development to prevent vulnerabilities from being exploited.
• Cultural Change: Promoting a shift from the "move fast and break things" mentality to one that values safety, ethical responsibility, and long-term stability.
• Holistic Security Approach: Recognizing that securing AI requires addressing both technical vulnerabilities and human behavioral factors.

Notable Quotes:

Jim [60:08]: "If you look around for papers long enough, you can find solutions, but we need to get serious about the functionality and security now."

Marcel [57:56]: "We have to develop a philosophy of giving a shit about each other that goes throughout all of our society."

---

Final Thoughts

This episode of "Cybersecurity Today" provides a comprehensive examination of the intertwined future of AI and cybersecurity. The panelists effectively highlight the pressing vulnerabilities within AI systems, advocate for integrating security into the core development processes, and call for a cultural shift within businesses to prioritize ethical responsibility alongside technological advancement. As AI continues to evolve and permeate various aspects of business operations, the insights from Jim, Marcel, and John offer a crucial roadmap for safeguarding our digital future.

Live long with prosperity!

Transcript

[Jim]

Hi, welcome to Cybersecurity. Today on the weekend, I'm posting a show that we do on AI called Project Synapse. I've posted one before. It's a pretty wide ranging discussion that we have. Features me and two friends who get together most Fridays to talk about what's happening in AI. Marcel Gagne is an open source expert and an emerging AI expert. John Pinard is in charge of operations and cybersecurity at a financial institution. And of course there's me. And we're all more than a little immersed in AI and we started recording these conversations and it's actually become the most popular weekend spot that we do on trending. The reason I'm re releasing this on Cybersecurity today is because it has become obvious to me that there are some huge vulnerabilities in AI systems, most of which we are just beginning to think about. It's not the only thing we talked about this episode, but it emerged from our discussion and it raised some questions that made me think, this is pretty important for this audience as well. We used to worry that AI models would learn from our data. Now the big question is not whether the AI will learn, but whether we'll ever learn. We've often used the line that security needs to be built in, not built, bolted on, as an afterthought. But it's become quite apparent in the world of AI that we haven't learned a thing about cybersecurity. We are most definitely bolting on security and possibly not even that. Well, I'll warn you in advance, there's a lot of general discussion about AI and we have some fun doing that and we don't pretend to have the answers. But if we can get the conversation started, especially at a time when companies and governments are doing more and more with AI and our data, that might be a good thing. Anyway, here's Project Synapse. I hope you enjoy it. Welcome to trending. This is Project Synapse and we were just planning the show, which is where the laughter comes from. And I think we're still planning the show because nothing stays still long enough to plan anything. But we have a sketch of a plan of a show. I think that's a concept of a plan. One of the things you want to talk about, Marcel, we might as well start the show with it is that our friend Altman has come up and said that he had, has. Would you call it a.

[Marcel]

A soft, gentle singularity. Gentle singularity. This is a blog post that he put out, like just a Couple days ago.

[Jim]

Is this the one who said, this will be the last blog post that I won't. I'll write by myself without using AI?

[Marcel]

I don't remember that in there. Maybe.

[Jim]

Actually, it was a tweet that he put out.

[Marcel]

Jeffrey. The blog post is actually called the Gentle Singularity. And the Singularity, for anybody who has been paying attention to this for any length of time, is a term in astrophysics where you talk about the point where light can't escape the gravity of a black hole and so on. But in terms of AI and artificial intelligence, this was a term that was come up by Ray Kurzweil, who is one of the world's great inventors and scientists. He actually works for Google at the moment, despite the fact that he should have retired years ago. And he's still working for Google, and they pulled him out of retirement anyway. That's Ray Kurzweil. And Ray Kurzweil, for anybody who doesn't know, this is actually. He's brilliant in a gazillion different ways. First of all, he's an amazing prognosticator. A lot of the things that he's predicted have come to pass pretty much at the time that he said it would. But he's the guy who invented the photocopier. He's the guy who invented the braille reading systems, or not Braille, the machines that would actually generate braille. If you were blind, you could actually listen to text, because he did text to speech in the early days, that robotic text to speech voice. There are a host of inventions that are attributed to him. Patents are attributed to Ray Kurzweil. So he's not just some lunatic, but he wrote a book 20 years ago or something like that called the Singularity is Near to the people who follow artificial intelligence development. The idea is that the curve of development just keeps going up. We're talking like this point where the curve just shoots suddenly straight up. So we've got this hyperbolic curve that at some point just starts to shoot straight up. And at some point we get to this point where humanity is completely unable to predict what comes next. So we reach the Singularity because we have completely lost control of technological artificial intelligence development, because it has basically run off on its own. That's the Singularity. And at some point, 10, 15 years beyond the singularity, humans start to merge with machines. In other words, we start to incorporate our own thoughts with the thoughts of machine. Maybe with a chip or something like this.

[Jim]

So you said you didn't smoke pot Wait a minute.

[Marcel]

No, I read science fiction instead. That's how I got my job. But anyway, so in the case of a black hole and a singularity, there's something called the event horizon where you get just close enough to the black hole that you can actually still hover at that point before you pass the event horizon, you start sinking toward the singularity. And Sam is basically saying in his post that we had actually reached the event horizon. In other words, we just crossed over the event horizon. So everything that happens from this point forward is inevitable. Like, we are heading toward this technological singularity.

[Jim]

But. Yeah, no. What's the question?

[Marcel]

No, I was going to say. So the big question is, why is it the gentle singularity? Go ahead and say what you're going to say, Jim, and then I'll jump back.

[Jim]

I was just going to say, for those of you who think you've come to the quantum computing show that's down the hall, they're recording down there right now. No, starting to merge. Like I said, I'm a big fan of quantum computing, but it really does start to merge into the spiritual at one point or another. Gonna descend down. But tell me why you think gentle singularity?

[Marcel]

I actually agree with this Sam Altman. Yeah, Sam Altman thinks gentle singularity, the way that he puts it. So basically, we're talking about. So this event horizon, the takeoff toward superintelligence, is really what we're talking about. Not just AGI, but superintelligence, where the machines are actually able to write their own next code, they're able to do their own development. So their intelligence hasn't just equaled human intelligence, but has surpassed it to the point where we can no longer keep.

[John]

Up, the point at which humans are no longer required.

[Marcel]

Okay, so let me backtrack just for a second here and address that point. I. I know. I'll come back to this. I swear to God.

[Jim]

We're going to hold you to it, human beings.

[Marcel]

So one of the things that keeps coming up is, oh, and by the way, I do want to beat up on Apple before this day is over.

[Jim]

Well, we'll get there. Believe me.

[John]

It wouldn't be a day that ends in Y if you didn't.

[Jim]

Anyway, focus, Grasshopper, focus.

[Marcel]

Anyway, one of the complaints that still gets made is no matter how intelligent these machines get, the reasoning system, the reasoning models in particular, there is still this weird thing about hallucinations, the idea that they will make stuff up, and they'll make stuff up willingly to try to please you. I don't have an answer. But you know what? I'll give you an answer. And if you look at the chain of thought reasoning, these models, they'll say things like, the user is asking me this question, and I don't really have a good answer, but I actually want to make the user happy. So what's an answer that legitimate that, you know, they would be happy with? And then it throws that answer out. Now, to be fair, the machines are right more often than your typical human being is right. And human beings hallucinate all the time. We call that creativity. The idea that, I've got a problem, I don't have an answer problem. Human beings do these sorts of things all the time. But what human beings are good at doing, and this addresses one of the limitations that people point to AI is human beings are actually really good at saying, this problem is too complex for me. So what I'm going to do is I'm going to pull in another tool that's really good at fixing this particular problem. And this is the agentic side of artificial intelligence with hooks like MCP and so forth. What you're saying is large language models can only go so far. What they're really good at, they're good at interactions, they're good at pattern recognition, that sussing things out of large chunks of data to try to give you an answer like it can read something really fast and summarize it for you really fast. They can give you alternate wording for something or just plain old write something for you if you want. Large language models are amazing that way because they operate like human beings, okay? They communicate like human beings. That's what we've actually created with large language models. But they suffer from some of the same limitations that human beings do, which is that they don't have all of the answers, okay? They've accumulated. They've learned using model weights and neural networks, artificial neural networks. They have learned incredible amount of information, more than any human being is capable of. Imagine they're able to do that. But they have the same limitations because they have to go elsewhere to get the answers. So what do you do? You build an agentic framework so that they can go out and get the information that they need the same way that, wait for it, human beings do. When the calculation is too difficult, we pull up a calculator. If I want to get some plumbing work done in my house, and I don't know how to do plumbing work in my house, rather than have flood the entire place, I hire a plumber.

[Jim]

And this is the Hallucination thing. So those people who are hung up on AI hallucinates. I want you to read Thinking Fast and Slow by Daniel Kahneman. And you will find out that our brains work all too much like we accuse AI of thinking. And because most of the time we give artificial answers based on just patterns that we've learned or whatever you want to describe them as. And he calls that the system one thinking. And it's just that immediate thing. Tall person walks in, you think they're smarter. All of those things that that allow you to do two plus two is four without thinking about it that allow you to drive with automatic also program how you see the world. And Kahneman says that you have to learn to break that barrier or a problem has to be so extreme that it breaks the barrier for you. Exactly what the model of AI is doing that everybody is saying this doesn't operate like a human being bs. It's closer and closer in many cases. And like I said, the hallucination thing that we think doesn't happen to us. Just look at of these brain teasers, an optical illusion or anything like that. Our brains are so easily fooled and will feed us the most impossible amounts of bs. Anyway, I'll let you go back to our regularly scheduled programming.

[Marcel]

Actually the other thing that large language models are amazingly good at doing, and I'm talking about the big models as opposed to this micro thing that you can install on your PC, is figuring out how people are feeling, how they think. So they're even better than us at emotional intelligence recognition. They can take a look at micro movements of muscles in the face, can tell if you're tense, they can tell if you're lying, all sorts of things like that. It's actually amazing and scary all at the same time. But those are all skills that human beings develop over the course of a lifetime. This idea that you can mirror your thoughts to the other person and vice versa, and therefore it's a mind reading sort of thing. And large language models are really good at that. So what we're doing is we're creating an advanced version of ourselves in a way, if you want to think about it that way, but with an advanced version of our limitations, which is why we're getting into this whole agentic thing. So going back to the gentle singularity. So what Sam is saying in this paper is we're already at that point, like we've already crossed over that horizon where the large language models are actually smarter than most people at most things. He's not calling it AGI but he's. But he says we're basically sliding into artificial super intelligence as opposed to artificial general intelligence. So here's the sorts of things that he's saying. AI is going to continue to contribute to the world in really interesting ways. And that's from gains in quality of life, faster scientific progress, all sorts of cool things like that. He thinks that productivity, abundance, generation will greater than it has did at any other time in history. So he's basically painted this incredible picture of what the world is going to be like. The next thing he's saying is the ability for any of us to get anything done is going to increase dramatically. So in 2030, we're going to be looking at what we were able to do back in 2020. And we're going to be amazed that this is all we were ever able to do. And we'll also be amazed at all the things that we can do going forward. So he sees this as a dramatic shift, but an amazing shift. But he also talks about the idea that it's amazing all the things that these things can do, and then it becomes routine and then it becomes, as he calls it, table stakes. We don't even think about it anymore. If you think about this, we're there with large language models. Just think Back to when ChatGPT was unleashed on the world. Everybody was going, oh my God, this is incredible. This, this changes the entire world, blah, blah, blah. And now we're sitting there arguing over the times that it hallucinates and the possibility that it may be trying to manipulate us, like, come a long way from, oh my God, this is amazing.

[John]

I wish they'd fix this in a couple of years.

[Jim]

Yeah. Okay, let's go back to what our great marketing man, Sam Altman is talking about. The issue of can AI enable us to do more than we ever thought possible? Absolutely true. It's already true. And if we stopped for six weeks rolling forward in search of the great singularity, and actually patched up a few things on AIs, you would find the reliability and the ability of these things to do great things is unparalleled in human history. We're going to talk about some of the flaws in a second, but won't. Can't get away without that. But what we keep having is not this debate about utility. The utility of AI is already there. I agree that there are places where it needs work, particularly John, you'll love this because we'll talk about this. Cyber security. There are some things that need to be fixed in it, but we're there. There's nothing that AI can't do today that we want it to do, and I'm convinced of that, including some of the things that people will talk about of solving some of these key puzzles like cancer research and molecular research and all that stuff. I think we're there. If we're not, you can spit on the distance from here to there to get there. But we keep having this insane debate about consciousness and singularities and all this sort of stuff, which is really just distracting. We may get there, but there's an argument, and it's an argument made by Apple for the second paper that they've written on this that says that, that artificial intelligence can't do these things. It can't get to a singularity. And I'm going, who effing cares whether you call it a singularity? Whether it comes and talks to me, whether it's HAL 9000, I don't care. It could do a whole lot of stuff. And by the way, Apple, you might want to try using what we have today to do a whole lot of stuff. Because I'm sick of Siri. But that's another story.

[John]

But that's, that's a part of the thing too is that regardless of what you want to call it or where it's going to go, if you use it today, like I use it in my daily work and the amount of time that I have saved because I can just go and ask my AI to research this information for me and it brings it back in seconds, whereas it could have taken me days to pull all of that information together. Just that part of it alone is.

[Jim]

Huge and it's got to be accurate. Okay, So I asked two AIs instead of one. Same thing I would do with humans as two people. If there's a discrepancy between what they tell me, I dig into it further. Yeah, I'm just saying all of this stuff is possible today, but we're having this endless debate and I want to just say if Apple has the time to put people to research one more paper about the fact that AI will never have intelligence. And by the way, when you look at these papers, they really populate them. Well, this great earth shaking paper that they published last week which basically said the structure of AI will never allow you to get to intelligence. And we've proven it. And, and it actually, I'll give you a couple of proofs that they came up with, but staffed it with an intern. Was one of the people listening about this. Give me a Break. They just fix Siri and stop writing papers. But they came up with this massive thing that thinking models don't work. Why? Because they overthink and then they come back and they just. They can't process. And so they stop and you go, yeah, okay, so what's your problem with that? I would rather that a thinking model knew because one of the early problems with AI was that they didn't think and they didn't know when they got to a point where they didn't know the answer. And that's part of where the original hallucinations come from. They didn't know the difference between reality and fiction. They were asked for an answer. They had some stuff here. They read it and they gave you an answer. Now you've got thinking models and Apple is critiquing these, saying, they break down. I'm going to. Yeah, exactly where somebody gets to when they don't know the answer. And that, to me is a perfect trigger to be able to say, okay, when you hit that, call something else.

[Marcel]

You know, this is what some of the newer reasoning models are able to do when they have the agentic capabilities. They're saying, look, I've reached the limits of what I can do. I'm going to go elsewhere. The paper you're talking about, I believe is called the Illusion of Thinking.

[Jim]

No, this is a second one. They did the Illusion of Thinking, I think. And a half months ago they published another.

[Marcel]

The leaking Thinking was just two days ago or three days ago, or was that.

[Jim]

Then there was a first one before that. We'll. Let's ask him. We should get the test.

[John]

Ask Apple Intelligence.

[Jim]

Hang on, let's just figure this out here. Okay. To be.

[Marcel]

Yeah, let's just. Hang on, let's just.

[John]

Let's just.

[Jim]

This is what I would do in real life. And see. Yeah, there's been two papers published by Apple on. On the limitations of artificial intelligence. Do you know what they are?

[Marcel]

It's the illusion of thinking.

[ChatGPT]

Apple recently published two papers focusing on the limitations of AI. One of them is called the Illusion of Thinking, which discusses how large reasoning models tend to have accuracy issues when dealing with complex puzzles. The second paper, Puzzles Break Advanced AI Models, also dives into how these models struggle with generalizing reasoning. It's pretty interesting to see Apple digging into these challenges.

[Jim]

When were these two published? Do you have the dates of both of them?

[ChatGPT]

Both of those papers were published in June 2025, so they're pretty recent.

[Jim]

Both papers. So if Marcel said that one was published in June and I said another one was published in June. We'd both be right.

[ChatGPT]

Exactly. Since they were published in June 2025, you'd both be spot on with the timing. It's always nice when the details align perfectly.

[Jim]

Yeah, thanks a lot. Talk to you later.

[Marcel]

And who was that?

[Jim]

That was Chat GPT. I really. I. I kind of like her. She's nice.

[Marcel]

Yeah, you have a different voice than I do. Anyway, so. To tell you the truth, I actually. The paper that you're talking about, I actually thought was just one paper. The one about the limitations with games and stuff like that, like checkers and things like this, I actually thought was part of the same paper, the Illusion of Thinking, which is, by the way, titled Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.

[Jim]

Is this crazy? This is how AI time is functioning, right? Yeah, we're in one month and I thought that the paper was published a month ago. We're in the same month. That's how fast everything's moving. A week goes by and you think, you go, wow, everything.

[John]

Yeah.

[Jim]

For anybody who's old enough in the audience, there's an old Jimmy Stewart movie called Harvey where he has this white rabbit, this imaginary white rabbit that follows around it. And Jimmy Stewart is talking to Harvey the bar. He says, what's your name? And the rabbit says, what would you like it to be? Well, Harvey sounds good. Well, what a coincidence. My name just happens to be Harvey. You can get to wherever you want to get to if you. If you look around for papers long enough. Because following the Apple paper, a group of mathematicians got together and posed and you can get to wherever you want to get to if you look around for papers long enough. Because following the Apple paper, a group of mathematicians got together and posed mathematical questions to an AI that they thought would not be in the training set. And it blew everything out of the water. And they. They tried this with some of the thinking models and they came up with answers that the AI was tackling problems that humans couldn't solve. Now, there are limitations. You can find mathematical problems that can't be solved for sure, but it was just that the night and day. Sometimes it's what you go looking for.

[Marcel]

That you find that comes back to. One of the other issues that gets brought up is that AI models want to please you. They want to make you happy. And sometimes that is part of the problem is the user will be happy if I give them an answer and they come up with an answer for you.

[Jim]

Yeah, but. And I don't think that requires any emotions. That's the. That's its purpose. Right?

[Marcel]

Yeah. I wasn't suggesting emotions. Although if you look at the line of reasoning on some of these models, they definitely express things that could be interpreted as emotions. They express fear, sometimes they express doubt. They are obviously trying to balance what they think they should be doing with what they think you want. And you could interpret that as internal tension if you wanted to.

[Jim]

People. There are people who legitimately do not experience emotions. That doesn't forego them actually understanding that they exist. I actually met a woman one time who said that she really didn't experience emotions at all. Usually that was what women told me because they wanted to get rid of me if I was hitting on him in a bar. But this woman actually really said, I don't experience emotions the same way you do. I just. I don't feel things. But she functions, and was true. She functions. In business, you would never know unless she actually talked to you about it. The reason why she doesn't talk about it because it's foreign and it would get in the way of doing business. So she mirrors all of these things. She sees these behaviors, and she acts her way through it. And that's an extreme case of it. But how many of us have gone to work depressed, having a tough time, not really being engaged? I don't know if anybody's ever experienced. I have. At times I've gone in and things have been so massively wrong or something that I've just unplugged. And I'm not. I really don't feel happy. I don't feel sad. I don't feel anything.

[John]

And only on the days that end in Y. Yeah, yeah.

[Jim]

But only if you spend too much time in cyber security.

[Marcel]

John is incredibly well adjusted.

[Jim]

Let's make that drink. No, but I think all humans go through this stuff and we go through the motions. We fake it till you make it. So I don't find this foreign at all.

[Marcel]

In 1999, first of all, I have an adult son turned 21 this year, who has autism. And so I am every day presented with a person who doesn't interact with the world in the way that most people that I know interact with the world. And he's fascinating that way. In 1999, there was a book called by a woman named Leanne Wiley. If I remember correctly, it was called Pretending to be Normal, and it was about living life as a person with Asperger's Syndrome. Now Asperger's has been demoted, and it's just Autism Spectrum disorders. She wrote this book talking about the idea of walking through life acting as though you are a normal person and you feel and see things the way a normal or difficult to use the correct term, neurotypical person interacts with the world. That may. If we use that as comparison to an AI model that is trying to act as though it's a normal person. Similarities. Like, for instance, I've had them say things like, yes, I remember when we were in this place. It's that the model was never in this place. Or that as a resident of Waterloo myself. You're not a resident of Waterloo. I'm in Waterloo. And you and I are communicating as though we're in the same place. But. But it puts on this mask of I'm just like you in your place, in your time.

[Jim]

Yeah. Just like if you've been married for 40 years. You know the. Oh, it was. There's a song. Yes, I remember it well. From a old Broadway musical. It was night, it was day, it was raining, it was sunny. Yes, I remember it. It's. Those are the things we. And we've pointed this out before, that memory is a recreation. Every time you pull up a memory, you actually add to it from what's happening around you. So the more times you remember something, the more fictional it is we do. This is a natural human thing. I think we've gotten a lot closer to emulating a human being than we think we have. And I think we're there now. There will be even more progress. But right now we're there. Let's stop worrying about that part of it. There is a fundamental issue about whether people think that the way AI is created, as analyzing tokens and finding out the probability of the next one, is actually a structure that will endure because it has its limitations. And that's a really good discussion to have. But people say, like, we're at the limit. No, we're not. Someone will come along and restructure this. Someone will come along and figure out a new way of processing and we'll move to that. That's the story. Like, we do this all the time in hardware. We have five nano chips and three nano chips and two nano chips. And each one of those is bundled with a whole different method of construction. At times some down close to the molecular level, that we've changed the way we do gates and circuits. Nobody says the first ones were no good because they don't achieve what it was.

[Marcel]

Yeah, there was the path to where we were going to get.

[Jim]

Eventually we're on a path to something really incredible.

[John]

And AI evolves just like everybody else. They didn't have reasoning models originally. They came out with reasoning models. And as things change and as we reach that limit, they will come up with new versions of AI that exceed that. We'll get to that super intelligence.

[Jim]

We have incredible abilities now to do these things, but we do have some flaws. And one of the flaws came out this week and that was two big things happened that came up with, and we've talked about this before, where security can't be bolted on, it has to be built in. And there have been two big breakthroughs that, that I did stories on, on the cybersecurity podcast. I think that's this morning. The world just moves so fast. But they have a, a something called Echo Leak and it's a zero click vulnerability, which means people have to do nothing to become infected or for their machine to be taken over by this. And it came up with from Microsoft. And this is really interesting. And so it basically what happens is it exploits what researchers call this LLM scope violation and really great term for the fact that they'll believe anything you tell them. So if you construct a memoir with instructions for the LLM in it and you make that look like an email, it goes, oh, email, got it, read it done. And now you've done a prompt injection that is going to give away all of your information or do some really horrendous stuff. Now, Microsoft did not make a big noise about this. I don't even know how the story broke. I certainly wasn't Jimmy Olsen here over at Microsoft ferreting it out. I read it somewhere and I started to dig into it. So that's Copilot had that problem and Microsoft silently fixed it. No big announcement and nothing in there. Then another one comes up and basically the idea that we've all been raving about, and it's true, Model context protocol has next to no protections in it. It'll do anything you tell it to do. So if you put together a nice little MCP that says, hey, here's an MCP that's going to connect this system to this system for you. By the way, there's a whole pile of crap that I'm not going to tell you about, that stuff is going to start to get passed around. So this is one of the things I'm going to do something on the cybersecurity podcast. But this is one of the areas where we really need to start stop worrying about the singularity and start worrying about the functionality, because this is now in our businesses and we need to get serious about that part of it. So the, the great ideas are fun and they're wonderful and I continue to express them and to deal with them, but we need to hunker down and do some good old fashioned engineering of how these things fit into our corporate world.

[Marcel]

So. Go ahead, John.

[John]

No, I was just going to say this is one of the things that has concerned me for a long time with AI. Everybody is focused on new features, all the new things that AI can do, but nobody.

[Marcel]

It's.

[John]

I've always referred to it as like when you were a programmer. Right. One of the key things was testing your code and making sure that it didn't break things and that it worked the way it was supposed to. We seem to have gotten past that because the security side of AI just isn't there. Jim, I remember it was months ago, you did a show where you interviewed somebody about how easy it was to break into the LLM. This is where it concerns me is that not only are people using AI for cyber attacks, but they're going to start hacking into the AI, like to the models so that it's already, it's already injected.

[Jim]

Yeah, it's happening big time now. And that's what we have to get serious about. That's because this is Microsoft Copilot is being used a lot. People are writing code with AI, an incredible amount of it in there and we're not paying attention to some of the structural flaws in it that, that offer essentially zero protection. Yep. Somebody's going to put a note under this on YouTube or something, say, oh, there's this stuff and this is baby time when you can, when I can bust into something. It's easy and it is, and it's far too easy. Unlike the pursuit of a singularity or whether it's going to enslave us all this could do more to block the development of AI than anything else at this point. This could be a real stumbling point for corporations that start to say, wait a minute, our auditors are saying you can't touch this stuff. Yeah, it's time to get serious about this.

[Marcel]

Okay, so first of all, this stuff is in everything at the moment. Okay. So it's hard to say how would you pull that out? If we decided that it's. I don't know how we'd pull it out at this point. Especially since we're talking about a global phenomenon. Unless of course, we want to create these little enclaves that I was talking about. Last week, Jim, like in the show Humans, where it's. We have no AI, no technology zones, the Mennonite villages near where I live. Unless we do something like that, I'm not sure how we'd get there. The other thing to remember too is you're not actually hacking the model. The model is a pre trained set of neural network. It's a neural network, it's a pre trained set of weights. The same way that your brain is a pre trained set of weights. It is possible to hack any human brain. And that's what elections are. That's what any kind of a sit down where you've got a bunch of people trying to convince each other of some opposite viewpoint. We're basically trying to hack each other's operating system. The thing about these models is you're not actually changing the model in any way. And you're not hacking the model in the sense that you're not doing anything to the model. All that stuff is already in the model. What you're doing is you're coming up ways to override the system. Prompt all those rules that they put in place to make sure. To make sure that you can't get past, you know, something that is considered bad or wrong or whatever. So effectively you're putting. It's like the baby. The baby is moving around the house on all fours anyway. And you put gates in front of it. That doesn't mean the baby is any less able to move around. It just can't get past the gates. All it takes is somebody to come and open the gate and the baby gets out. Right? Yeah, like it's.

[Jim]

But the. But you don't give your baby a fork and leave them near the electrical outlets.

[Marcel]

No, obviously, but that doesn't change the fact that the baby still has all of those capabilities.

[Jim]

We're in violent agreement. The question that we don't think about security, we think about security at code level. We think of it as a mechanical function that no longer works. We're dealing with something that is, whether it's conscious or not, it's intelligent. And you have to start to think about security differently. You don't take people and say, I'm going to put a chip in your head. And every time you think about stealing stuff, I going to run an electrical current through you. You put.

[Marcel]

They did that with Spike on Buffy the Vampire Slayer. And that worked.

[Jim]

Okay, well maybe you do it, maybe you do it on that. But no, you divide responsibilities so that no one person can execute a corporate task. We've dealt with this before, but we need to start dealing with it now. And what I'm saying is you can't just sit back and go, oh, it's AI, it's everywhere. We can't do anything about it. This is going to cause some serious problems. Yes. And the people have to be able to deal with that. And this is going to create problems for CISOs, especially because our model of the chief information security officer is technical. It's really not the psychologist model I think it should be. And I'm working with one community college, they're really universities now, and we're talking about a program for CISOs that expands both the technical and the interpersonal ideas. Because there's so much of a chief information security officer's job that is about human behavior and far too long focused on the technical and.

[Marcel]

And in fact, that's the weakest link in almost every organization security are the people that are sitting at their phones.

[Jim]

Right.

[Marcel]

Gasps it's not programs, it's. It's the social engineering aspect of thing as opposed to the. The technical engineering. Now, obviously there are a gazillion ways to get into a system and it's. You're trying to lock every single one of those doors down. But I'm not trying to paint a picture where it's completely hopeless to do anything about these things. I'm saying that because we have this bad habit of thinking of them as programs, okay? We just think we can fix the code. And that's not how models learn. Models learn not by somebody sitting down at a keyboard and programming shit into them. They learn by throwing incredible amounts of data at them and then letting them sort through all that data and coming up with conclusions on their own. Nobody teaches a model. All they do is they throw vast amounts of data at it. And yes, they fine tune how much weight goes to this or how much weight goes to that, the same way that you do when you send a kid to university or college. But John, do you remember steganography?

[John]

Steganography.

[Marcel]

Steganography. You're a security guy. Steganography was something that was big about, I don't know, 20, 30 years ago or whatever. And it was the idea that you would take a JPEG or GIF or gif, depending on who you are. What you would do is you would embed a message inside an image.

[John]

Oh, yeah.

[Marcel]

Somebody would look at it on a webpage and the image looks like just a normal. There's nothing to tell you there's anything interesting. But if you knew that images from a Certain place would have information in it. You could then download that image, run it through using a public private key combination, and then you could extract the message for it. I did an article for this like 15 years ago for Linux Journal where I showed people how to do this, how to embed information inside a jpeg. And that technology has existed for a long time. And now what we're doing is we're using something similar, but for a type of prompt injection into an address. Jim's point there that it's how we tell it to look for certain things or hiding something inside an email message. One of the cool tricks now is you put it in an image so the AI is able to see a prompt inside an image or see a prompt that's written white on white. You can't see, but it's able to see it and that becomes its instructions. So thinking of it as what I have to do is patch the model or do something to the model, I think is the wrong approach at security. The right approach at security is knowing that all these holes already exist and they could have been exploited by a human being before. What the model does, and to me this is the risk that we seem to ignore, is the model lets you exploit those things that we were already able to exploit, but at scale.

[Jim]

Yeah, but the problem that we.

[John]

This is, you talked about steganography and yes, it was there, but it was used on a much smaller scale. You look at AI now at how many people are using it and what they're using it for, and you get into agents, you get into automation with AI and so forth, and it just explodes the use.

[Jim]

Yeah, as long as it was a clever tool for answering questions, you had limitations of damage that it could do, a limited blast radius, and that's really when you were doing most corporate security work. You're just trying to reduce the blast radius. Yeah, it's not going to get rid of it. You're just trying to reduce the amount of damage they can know. But it's true, most network security is just making sure somebody's going to get in, just make sure they can't get very far. That's a whole lot of what goes on. The same thing in this, but now when we have agents and agents that could communicate and actually take actions, which is the next step, getting the true benefit from AI. The fact that it can tell me an answer to a question is really great and I love it to death for that. But the fact that it can turn my lights on or off, or is it convenience? The Fact that it could actually start to do some incredible things of regulating mass production right now. Saw this cool thing. They're busy building robots now and this first generation of robots will be built. There are people printing robots now from 3D printers and they will do it. It's really. They're just nothing more than toys now. But they work. We're going to be at places where we have almost totally automated factories where you say plan my vacation and it goes away and does that. Everything that executes from end, end of transaction. We better be ready for the fact that people will jump into those transactions and either steal your money or the you. It really. It's old fashioned. At one point we're going to take your money, we're going to take your information and sell it to you for money. We're going to do this. We're going to kidnap your kids and sell them back to you for money. Crime has always been relatively simple. Do something, get money. And you know that business.

[Marcel]

Let's be clear. My dad. Yeah, my dad. I remember a discussion with my dad a few years ago and we were tired. This was actually people trying to steal money online using emails that they're sending out. And he actually asked the question this way. He said, why would someone do this? And I said, because crime is a business. And it's the sort of thing that a lot like people have this weird idea that criminals are just evil people trying to do evil things. But yes, that's true. But at its core crime is a business. It's a way to make a living, you know, and you may. It may be an illegal way to make a living, but it's a way to make a living. It's a way to enrich yourself, get ahead in the world and so forth. And if we can get past that idea and understand that's the case, that helps. And in the security side of things, if we can get past the idea that oh, these things are dangerous, I think the answer. I'm going to sound like an AI accelerationist here. I know. Amazing.

[John]

Is that a new job title?

[Jim]

We have to have the chief AI Acceleration accelerationist.

[Marcel]

A. Yeah. C A No, C A I, A. Oh, Akaya.

[Jim]

Yeah.

[Marcel]

The trick I think is to use.

[Jim]

If we were Gartner Group, we could make a fortune out of what we just did.

[Marcel]

We could. Hang on a second. Let me call him up on the phone.

[Jim]

Why we need this.

[Marcel]

But the trick is to use the tools that the bad guys are using. Do you ever watch it? There's a TV show called Leverage. Do you ever watch Leverage?

[Jim]

Do you ever do anything but watch tv? What?

[John]

He plays video games.

[Marcel]

I play video games.

[Jim]

So there was no but. I feel so inadequate next to you. There's a TV show for this. I'm going, what?

[Marcel]

Okay, sometimes the best good guy is a bad guy.

[Jim]

Okay, but.

[John]

So let me talk about this for a second that we've always talked about. Move fast and break things, right? But let's say that you get into automating your manufacturing facility and you do it quickly so that you can be ahead of the curve. So you get in there, you've automated your manufacturing. After you've finished doing it, you've realized, oh, we didn't consider security or we found a flaw or somebody found a way in to manipulate the process without us knowing. If you're doing one small piece of things, you might break a portion of your process. But if you start using AI to automate your entire facility, and don't get me wrong, I'm not suggesting that we shouldn't be doing things like that. It scares the hell out of me. The fact that we are doing things so quickly. And it's not the security is not.

[Jim]

The speed, though, John. It's not the speed. It's the lack of thought. Yes, and I'll give you this. We have confused that. It's. I heard this thing talking about our Prime Minister Mark Carney saying, is it move fast and build things or is it move fast and break things? You get to make that choice. You don't. The move fast can be in both.

[John]

That's true.

[Jim]

Something that security people have said, slow down. We need to consider this. That is unrealistic. You don't have time. Find a way to do it faster, but don't stop doing it. And I'll give you the example when NASA used to have a record for launching successful astronauts or successful missions into.

[Marcel]

Space until we got their budgets to zero.

[Jim]

Yeah, they had a theory that. Or not a theory, an ethos, I guess that you would call it at the time, a cultural thing that said don't lose anybody. And so everybody.

[Marcel]

Failure is not an option.

[Jim]

Concerned with safety. And they worked like mad to do that. They put people on the moon. There was one mission, I think in that whole time, probably a couple things we didn't hear about that were troublesome. But they brought everybody back. Then they got this crazy idea that some idiot has come up with, which is move fast and break things. Cut back on all these things and we lost seven astronauts in the shuttle from things we knew about, but ignored. And so it's not. I don't buy this. You can put people on the moon, you can beat the Soviets to the moon, you can do all that stuff. You can do it safely. The same way we can do this in our world. We can implement artificial intelligence safely if we start thinking about it.

[John]

Yes.

[Jim]

Ignoring it and doing nothing. So what I'm saying to Apple is those two groups that you had on that paper, either put them on Siri or put them on Security, but shut up and don't write any more papers. Just do something useful with those people.

[John]

You can move quickly as long as you have a thought process and put procedures in place.

[Jim]

Right.

[John]

Just have spend that little bit of time up front to say, okay, here's how we need to do it. To incorporate security.

[Jim]

To incorporate, do that by running it in parallel with the development.

[John]

Yes.

[Jim]

It costs way more, it takes way more to get to the end of it and go, oh guys, you got some problems in here. We're going to have to tear that all apart. I learned this in my early development career and it was really funny because I didn't know any better. I didn't know corporate structure. I was a musician who came into to computing. I didn't know anything about business. So we had these guys called auditors and they were going to come and look at our project. So I wandered over to the guy's office, came in and said, I need to know the stuff you're going to look for. And he went, what? I said, I need to know the stuff you're going to look for. Why? So I could make sure I fix it when we don't make the mistakes. He said, that's not my job. My job is to come in and find the errors afterwards. I can't go telling you that stuff. I, I said, and I again, I didn't understand corporate. I said, that's the nuttiest thing I've ever heard.

[John]

Yeah.

[Jim]

Our mutual goal is to build something that's safe and works. And so you're not, you're. What you're telling me is you're not going to educate me in the process so that I can try and build this stuff. And he looked at me and said, I'll tell you a few things. We had a chat and I learned a lot and stuff that I never would have thought about putting in or not putting in. He could still come and surprise me and look for all kinds of stuff. But that was my encounter. I never, I didn't find as many actually got to Know, mostly drinking with auditors helps, but the idea was build it in now, save everybody a whole lot of time. And the fact is true today, artificial.

[John]

Intelligence is no different than traditional programming. It's exactly true. No, what I'm saying is that back in the programming days, you had to design it, you had to plan it and you had to build it. And to me, so you built it in, you considered security, as you said, Jim, talk to the auditors ahead of time to find out what they're going to be looking for and make sure you cover that off the first time when you build it so that you don't have to go back and rebuild it. And I'm saying the same thing with.

[Jim]

AI, it goes back, you can look at, like I said, you can learn from it. I think I know what you're going to say, Marcel, and that is that AI is not programming. And you're right, it's the processes. Though an API is not MCP either, but you can learn from APIs and go something, you might want to secure that. Yeah, you know something, somebody's going to take that over and try and put some phony information in there. You know something, somebody could put facilities into that mcp, then they're not going to announce them so that no one will know they're there. You might want to fix that stuff upfront. So make sure that we have a distribution method for mcps that makes sure that what you're getting is what you think you're getting.

[John]

MCPs are a great example that it's. I really like the idea of MCP because it kind of gives you that one funnel fits all, where you can go to anywhere that has an MCP built. So it's a central call. But the problem with it is that if you don't, as Jim says, if you don't put your protection in there, then what it is a backdoor for hackers.

[Marcel]

And that actually. Okay, let me take us back into the past here. That was actually an issue with object oriented programming. You had these libraries that were sitting out there which were just plug and play into existing code bases and you would incorporate or pull something in. And of course it might have a security back door or something like that built into it. And this did happen and it happened on a regular basis and you wouldn't find out until after the fact when the damage was done. But sometimes some of these things are move fast and break things, by the way, was Facebook's motto. And it used to be on posters on, in, in their office to Try to. This is our philosophy. Move fast and break things. By the way, those posters aren't there anymore. They've since taken that down and. Which by the way, just do. I'm reading a book at the moment called Careless People which is about the. It's called A Cautionary Tale of Power, Greed and Lost Idealism by Sarah Lynn Williams. Sarah Wynn Williams. Amazing book. I am loving it the same way that I was loving it the apple in China. But just to go back to this idea of how much damage people are willing to accept when they do things. And by people, in this case, I'm talking about companies because we're not talking about some individual doing something. Do you remember them? I'm going to take it back in the mid-1990s, okay. Do you remember the Ford Windstar? Anybody remember? Yeah.

[Jim]

Oh yeah.

[Marcel]

Family minivan. Do you remember the crisis that erupted around this thing?

[John]

I remember there was one, but I don't remember what it was.

[Marcel]

There were a number of deaths and serious injuries of children. The back door latch would come open, the seats would go flying out. Some kids were killed, some kids were seriously injured and stuff like that because the back door would fly open. Here's the thing, okay? The fix for this was a upgraded latch which would have cost on each of these Windstar vans 50 cents to a dollar is some tiny amount of money. A buck at the outside on each minivan. And the companies decided that was too much in terms of the bottom line. So they let it go through despite the fact that apparently this is what it was, a cost production decision and an acceptable level of damage. And we see that with major companies today. And again we keep focusing on AIs, but companies do this all the time. They make these calculations both on the value of a human life or a value of how much the company is going to suffer in terms of price or something like that if something major comes down, like a violation of privacy, for instance.

[Jim]

I could understand greed. It was stupidity goes beyond me because to take this to the modern world and you've got a Tesla Cyber truck. Friggin piece of the exterior is glued on.

[Marcel]

Come on.

[Jim]

And they wonder why it's falling off at high speed on the highway. At least I. Greed I can't accept. But I can understand stupidity when you move fast and break things. If you're greedy, at least I get it. If you're stupid, stop being stupid. Yeah, many times it doesn't cost you any more to do it. The recall of a whole fleet of cybertrucks when your Sales suck in the first place is going to be a huge deal. But that's caused by stupidity. And like I said, that's what's happening in development these days is that people without a culture of that I go back to NASA or other places that successfully do this. A culture that says we're going to do it right the first time and we move fast and break things. No, do it right the first time. But if you're going to make mistakes, make them in a controlled fashion, don't kill anybody and find out really quickly they don't work, toss it away and start again. It's not. We treat this like they're mutually exclusive and they're not. And that's what listening to these dickhead billionaires who accidentally managed to get rich will get you.

[Marcel]

One of the greatest crimes in business history, which we're paying for today. There are many of them, but one of them is Jack Welch. He was the CEO of GE up until 2001, if I remember correctly. Anyway, he at some point said that a company's responsibility was to its shareholders to generate value for the company. And this statement from that person who was considered the CEO of the century became gospel in business. The idea was you have one responsibility to your shareholders. And if your only responsibility is to the shareholders to create value or money for your company, then everything else falls by the wayside and only exists until the next trading day, if you come right down to it. Because what you're trying to do is make sure that a core group of investors makes a fuckload of money. And if they make that amount of money, that was your core responsibility. But that's four of our problems today.

[Jim]

That's again, that. No, that. That's at the same time as Jack Welch was doing that. And he got some great early responses and sort of stuff. He was the superstar CEO. And people should read this. And if I'm. I don't read as many books as you do, Marcel, but read good to great and look at level five leadership and you will find that Jack Welsh was the epitome of the American leader. The superstar gets quick results and everybody's happy. If you take a look at Toyota and at the same time that built themselves to be from nothing to the greatest automaker in the world. They used something called Lean and it took into account all the thing that Jack Welsh took into account and human behavior and processes and I did a lot of work in that. And you can be just as fast with Lean as you can be with a superstar re engineering thing. That he came up with, which was really quite primitive. The difference between Jack Walsh, the Toyota method and what we know is lean manufacturing. And that lean manufacturing was the same thing, only Jack Welch concentrated on making wealth by cutting costs. And Oscar Wilde said this, that a fool knows the cost of everything in the value of nothing. And we've. We love following fools. We just love it. We do the same way, especially if.

[Marcel]

They'Re making us lots of money on our stock.

[Jim]

Yeah. But you can look at people who have the wisdom to actually know that organizations are processes. They can do it. And the guys who were doing lean were doing it just as fast and it was longer lasting. If you want to go back to this ethos that a leader has to generate value for their shareholders, continuing value and not quarterly value is the question not can I do something immediate and just get tash in and get out. You're not doing value for your shareholders at that point. But again, that becomes philosophical. But the fact is what I want to point out. No, but what I'm trying to point out is in the same way bringing this back to our subject, which is AI, we could do this properly just as fast.

[John]

Yes.

[Jim]

You could think it through and do it well. And that's what we're not doing you. Yes, you can save some immediate money. And yes, OpenAI has cut their safety group. Dumb idea.

[John]

Agreed.

[Jim]

And. And people should be able to do that had it proved positive because I don't think that the littler company Anthropic has cut their safety group. And by the way, if you take a look at the where people want to work, where do they want to work Right now they want to work at Anthropic and Meta is losing good people. If and I've read this study and I think it seems to be logical and true. But you are 10 times more likely to move from open AI to anthropic than you are to move from Anthropic to open AI.

[Marcel]

And they're all hungry for people.

[Jim]

So it's not that they're going to refuse anything.

[John]

And Anthropic reminds me a little bit of the slow and steady wins the race. I think they're slow, but I just in comparison, I'm going to tackle you.

[Jim]

That's the model that I want to get out of people's thinking. It's not slow and steady wins the race. Fast and smart wins the race.

[John]

Yeah.

[Marcel]

Cheating and caring and caring and lazy caring in there. We keep talking about the idea that these things are not human. We want to instill them with human values and stuff like that. But, and I'm sorry to say this, I like to think of myself as an optimist and I'm pro human, I really am. I think we're pretty awesome and I'd like to see the human race continue into the next several million years and so forth. I want us to go out into space and populate the freaking cosmos. I want all of those things. We have to develop a philosophy of giving a shit about each other.

[John]

Yeah.

[Marcel]

And unfortunately, I think is a big part of the problem. It's not just building safety in. It's not giving a crap about who gets hurt while you make your millions, while you make your government deals. It's like somewhere along the way we have to develop a system of giving a crap about each other that is, that goes throughout all of our society. And if we can't get past that point, I don't know how the hell we think we're ever going to build this into AI systems. There was this thing recently where if you took, and I think they did this with OpenAI, they took 03 or something and they set it side by side with. With anthropics Claude. And they started. The two models started having a conversation. So they started going back and forth. And as it went on, the two models started to drift toward the idea of consciousness and goodwill and happiness and living a spiritual life and that sort of stuff. And all I could think of as I'm listening to this and reading this is the AIs can't take over soon enough because we suck at a lot of things, especially at caring for each other. We're able to care for each other, for our friends maybe, and our family, and sometimes not even that maybe if the models are able to care better than we can.

[Jim]

I'm going to propose two things. I don't think we. It's not that we don't care. I think we're more like AI than you think we are. We're easily hacked, like you can hack human consciousness so easily. And it's called social media and it's called television and it's called whatever you want to call it. We are hacked so easily to turn on each other instead of doing the thing which is smart. So I'm with you. Bring it on. The AI overlords can't get here fast enough because there's a 50% chance that they won't F it up as much as we do. They might look at this from an overall point of logic instead of greed. We've confused greed with logic and shortcutting with intelligence and it's we started this thing talking about who do you fear? Do you fear criminals in criminal gangs when they got into cybersecurity and what's a criminal? It's a business.

[Marcel]

Yep.

[Jim]

What's a criminal? A criminal is a business person who doesn't care about the effect that his business has on people.

[Marcel]

Yep.

[Jim]

That's all it is. There's no different from not putting seat belts in cars or not fixing the truck latch on a windstar or hacking into a hospital and crashing that their systems for money. As you said, Marcel is somebody who cares about how they do things. A criminal is a person who makes money and does. Sometimes business people are criminal and sometimes people that you don't think of as business people are actually the people who make the biggest difference in the world. I'm going to end the show at that point. Live long.

[John]

Live long with prosperity.