
Unveiling the Complexities: The Dark Side of AI and Its Real-World Implications In this episode, explore the intricate discussions surrounding AI with experts Marcel Gagné, John Pinard, and Jim Love. Dive into contemporary understandings of AI, its...
Loading summary
Jim Love
Many of you seen Project SYNAPSE before. You've seen this discussion with Marcel Gagne, John Pinard and me as we talk about AI and how we use it in our personal and our work lives. And one of the things that we wanted to talk about, although all of us are positive about AI and its usage, we're not. We're not what you'd call doomsters by any stretch of the imagination. But we, we are people who go into this with our eyes wide open. And one of the things that I wanted to have a discussion about was what I've called the dark side of AI and Pink Floyd fans out there, forgive me, by the way, there is no dark side of the moon. Just in case anybody out there has been fooled back to reality. We wanted to do something on the dark side of AI, not so we can fear it, but so that we can plan on how to handle it. But as I discovered, there's a bigger discussion to be had on this and one that we should be all having. There's much more to it than we could simply get to in an hour and a half or two hours that we spent on this. So I'd like to invite you to join that discussion. You can reach me, Editorialech, Newsday CA. You can find me on LinkedIn if you're watching this on YouTube, you can add your comments under the video that's there. If you check the YouTube notes, or if you just go to technewsday.com or technewsday CA and then click the menu for podcasts, you can find the notes from this session. And when you do that, you'll see an invitation to our Discord group and you're welcome to join us there to continue this discussion through the week. Whatever works for you. Here's the discussion we had. Hope you find it interesting. I think there are three areas. One is the AI misbehaving in one way or another because you could jailbreak it. Then the second one is the AI used as a tool and the AI can actually be used to do damage. And the third is, if we're going to operate this within our production environments, how are we going to protect our data and our processes? That was the three organizing principles I put together. Is there anything I'm missing on that?
John Pinard
I do want to spend some time talking about the far fetched thing, and you can think of them as far fetched as well. One of them dovetails in what you said, which is the AI misbehaving. This idea of the AI trying to break out or the AI purposely lying or the AI purposely deceiving people. Which of course gets us into the science fiction scenarios of Skynet. I guarantee you there are people out there who. That is their big thing. Talking about things like disinformation. Personally, I think disinformation and misinformation is one of the greatest dangers, because you're able to do that at scale somehow. That's one of the ones that almost nobody thinks about. That is, to me, the primary danger, and yet the one that gets the least attention. What gets the most attention is Skynet. I do want to touch on those things. I want to talk in the speculative risks to some degree. Obviously, the ones that you've mentioned are real and legitimate, but they're not sexy.
Jim Love
In terms of sexy, yeah, but. But I don't want to get into a hype thing. I don't mind talking about those, because what you're talking about is real.
Marcel Gagne
I think that people need to understand. Like when I was going through this, I was thinking about it from. And I want to implement AI at my company.
Jim Love
Yeah.
Marcel Gagne
And so I think one of the things, and it was in that final thought on the one that I sent, is that AI is a powerful tool, but it's not just plug and play. You need to look at the security. You need to look at. How are you going to deal with misinformation or disinformation? Are you going to put in special processes that say if you're using AI, you need to go through and do this to verify it as an example.
Jim Love
I totally agree with you, John. That's why I said your list really looked like a list that somebody had prepared. If I'm in a corporate setting and I want to implement, this is all the things I should do for implementing it so that it's protected, so that it has all of the aspects that we need for corporate setting? The idea that I was talking about was, hey, there's two ways we can use this. One is it can be used for evil, which is another thing we have to control. A lot of the criticism of Deep Seek is that it has no guardrails. It can be used for anything. The other piece of it is, can we get in and jailbreak these things? Poisoning and the fact that they lie. And that really does lead to where you're going, Marcel. And I think I'm in a disagreement with you, or maybe I'm not, but I believe this stuff is way more real than we think it is. I studied complexity theory for a long time and I. I Think we don't understand because these are neural networks and prediction machines. They'll never be able to think or do anything. I think they're. I think they potentially have more capability than we think. And when we add another layer to that, some of the science fiction stuff could very well come true. I. I listened to Mohat and he blew my mind in terms of what he believes is already happening. And he's one of those people who believes that when he saw the first indications of, remember the story he tells. What is the guy's name? Just. It's Mogadat.
John Pinard
Mogadat.
Jim Love
Mogadat. Yeah. But he tells a story of walking to his office and they had all these hands that were going to be gripping balls and they. And actually really is a tough thing to get automated hand to pick something up because if it even moves a quarter of an inch, you lose everything. So he had all of these machines that were trying to do this over and over again. He came by on a Friday night, walks up to his office and suddenly one of these things is picking things up perfectly. Each time, hours later, he comes back, they're all doing it. Was it conscious? No, but it has a behavior that is unpredictable, unexpected and. And independent. I. I think it, it happens. I'm going to sound like some new age or something when, when we feed the birds in the morning. The birds are starving here. I can just go over to Jim's place. These bunch of freeloads, I put the stuff out for the birds. One bird comes. No time for it to go and walk off to everybody else and go yak, yak. So I know they're not talking to each other. That bird flies away, but they start all peeling in. They are so attuned now. What is it? I don't know, but they have incredible.
John Pinard
Eyesight, for one thing.
Jim Love
Yeah, they pick up signals really quickly. Let's take that to crows. If you go and attack a crow, the whole flock will know who you are.
John Pinard
Yep.
Jim Love
All stuff we can't understand. And yet we think we're going to be able to understand these incredible machines we built. I think that's the ultimate in hubris.
John Pinard
Did you guys both watch the interview that I posted with Jeff Hinton? It would have been about, about a week and a half ago where Jeff Hinton argues that large AI models, frontier models, are already conscious. Yes, and I agree with him. One of my great obsessions beyond even this. And I've read, I cannot tell you how many books on this, listen to how many podcasts over the years. But I'm obsessed with the idea of consciousness and what the self means and so forth. And of course, I haven't helped myself by starting to meditate six or seven years ago. The. The idea of consciousness is an. Is truly an obsession. Hell, I have written a few blog posts about it. The idea that, am I the same person when I wake up in the morning who went to bed the night before? Did I go away? And basically there's a slightly altered version of me that's coming up the next morning. I think about this shit all the time. Like it's one of those things that's always in the back of my mind. And I think when we start talking about the idea of. Is the AI misleading us? And I said this on the very first Project SYNAPSE that we did. If we start talking about whether these things are misleading us, whether they are effectively lying, trying to bend the rules, or cheat in order to achieve whatever goal it is we gave them, then you either attribute the idea that they are to some degree conscious agents deciding what it is they're going to do, or they're not, in which case they're not cheating, they're not lying, they're just following a set of instructions within a set of guidelines. And we just haven't defined the problem sufficiently. So it's either one or the other. And I tend to lean heavier and heavier these days towards the idea that, yes, there is some kind of an emergent consciousness there. Not the same as us, but it is definitely an intelligence with its own goals internally. Before you get into the car and drive to Toronto because you've got a meeting or something like that, it's not like you went. I'm thinking that I should go to Toronto for a meeting that I don't know anything about next week because I haven't heard a phone call about. You wait until there is a goal put in front of you. You need to be at this meeting, you know, in a couple of days. That's when you put the plan into motion. That's when you start thinking about the contingencies and what you need to do to achieve that particular goal. It doesn't just come out of thin air. Something spawns that gold or gives it to you.
Marcel Gagne
There was a talk we had a few weeks ago about the AI model started to back itself up to a different server because it feared that it was going to be removed.
John Pinard
To me, that's exactly what I'm talking about.
Marcel Gagne
That, to me, is some sign of consciousness that it's it sits there going, ooh, I'm aware of my surroundings and what's going on around me. And that kind of ties in with the picking up of the balls that you were talking about. Did one learn from another? I truly do believe that there is some sort of consciousness in these things. Even from my usual go to of how many hours are in raspberry. When you say you're wrong, it goes and thinks and goes, oh, wait a second, yeah, you're right. And it comes back with the right answer. I think it's more than just this is what I mean about calculations.
Jim Love
Hinton had talked about this early in the game and I think he's thought it through. I think that was why he left Google. His theory is this is an alternate consciousness. We're trying to presume it exists, defining it as the way consciousness works for us. We don't know what consciousness is exactly. There's nobody out there who knows it. Matter of fact, watch a flock of geese, okay? And everybody will tell you the leader of the geese is there and they, you know, and they move, the leader moves. But this is some sort of conscious behavior. They're all looking at it, making a decision. Ridiculous. Why? Because if you model it, it won't work. But if you do assume that somehow programmed into them is the idea of keeping some distance, some signals that we don't see. Somebody created a program trying to put this together and recreated the movements of a flock of geese.
John Pinard
That's right.
Jim Love
Are the geese conscious? Is the simulation conscious? The answer is irrelevant. It behaves the same. And that's my theory with AI it's an attractive thing for me and I love the idea of pursuing it because like, humorous, all the ideas is. It's great obsession. It's a great thing to think about. People have thought about who we are and why we are and whether we're real and all those things for years. That's part of being human. From a practical standpoint, one has to presume if it walks like a duck and quacks like a duck, it doesn't matter whether you think it's a duck or not, it's going to behave like a duck. And I think that's the difference with AI is that we, we assume a simplicity when there's a complexity. And so a lot of people will poo. This idea of it tried to replicate itself. It doesn't matter whether it woke up and went, Geez, I'm HAL9000 and I'm going to think about moving on here. It did. And that's where Mogadat came into this. That said these things are already conscious according to his definition. And that presumes that they're going to have unexpected behaviors.
John Pinard
It's only been recently that we've decided that animals are conscious. It didn't happen that long ago. We were brought up to think that these were basically automatons. They were biological machines that had pre programmed responses. They're not actually playing, they're not actually having a good time. It's just a fight for survival. I've seen animals play with each other and they're definitely playing and having as living, breathing creatures. There is something that it feels like to be a cat, to be a bat. And just because we can't put ourselves in the mind of a bat doesn't mean that the bat isn't conscious. Consciousness is an experience that takes into consideration all the things that are around you. It's not just looking out into the world and creating a picture of it. The picture is everything and you're part of that picture as well. That's what's happening with artificial intelligence systems as well. Because there is a world out there. They are part of that world and it all comes together.
Jim Love
If there is this idea of emergent behavior, unexpected, unpredictable behavior, what does that mean to us who are trying to use AI today and tomorrow? One of the things that scares me is that there's not enough research into alignment. I've heard a lot of people talk about deep sea saying it has no guardrails. But then I'll read somebody hacked into the latest AI model, the OpenAI, the O3 model with its supposed sort of intelligent detection systems. The hack that the person did was absolutely simple. Some people have told me about how they've hacked these things and said you can't talk about it. This guy published it. So I'm going to say it. He just hammered away at it, changing a bit of each prompt, capitalized it letter differently and just in repeatedly hammered away at 03 and got in and got it to do something it wasn't supposed to do.
Marcel Gagne
Yeah, you know, and that's. We talked earlier about. I'd said one of the, the lines that I saw in an article I read was that AI is a powerful tool, but it's not plug and play magic. This isn't. You're not taking Microsoft Excel or Microsoft Word and sticking it in and just using it. You need to know what it's doing and what it's doing with your data. You can't just drop it in and Go, oh, I'm good. I'm going to start using it now. Because there are all kinds of things that can be done. You know me, Mr. Security. I have huge fears. You look at how fast these new models have come out, how fast these new tools have come out. Jim talking about the hacker that had gone in with the model. We don't know what we're going to find out down the road. It really makes me nervous. And I think people need to understand that when they're implementing these, whether it's for personal use, private business or corporate, you need to understand you're still responsible for the input and the output. So you need to make sure that you can explain or understand what you're getting out of these things.
Jim Love
Our old model of garbage in, garbage out served us well. Does it work now? No, there's no correlation. For instance, this person was able to hack into an AI model and cause it to do something differently. Following in our I could use it for evil, I could use it for purposes it wasn't used, or I could poison a process in it. I have heard people say that sounds like a rumor. I have listened to people who are embedded in this industry and who know well enough to know that it could pivot down to a single word in a prompt. Whether that's true or not, I don't know. But the amount of damage you can do in a model is not correlated with the amount of input you have to it.
John Pinard
Yeah. A few years ago, in the early 2000s, there was a guy who worked for one of the British banks or one of the British trading floors, Kwaku Abdoli or something like, I'm messing up the name. I apologize here. But he did some unsanctioned trades. He was given the power to go into the system and do some unsanctioned trades, and he lost the company overnight, something like $2 billion. That $2 billion wasn't just to his company, but to all the people who invested in this company as well. The lesson here is that those dangers, John, that you talk about of having some kind of oversight into what the model is doing and making sure, you know, the appropriate guardrails. Obviously the guardrails on this guy were not sufficient. The fact is, you give anything or anyone too much power without appropriate oversight, and bad things are going to happen. The real danger with artificially intelligent systems is that they can do these things at scale, at a speed that human beings can't possibly keep up with. And. And it's not that the systems themselves are any more dangerous. Than any human being, because a single human being can do an amazing amount of damage, given the position of power to do that damage. But the systems that we are creating are a thousand, ten thousand, a hundred thousand times more capable and faster than any human being. So if you give them that power, that's where the real danger applies. It's not that the machines are inherently less safe than humans. I would argue that the systems we have are actually less dangerous than the human beings that we've put in power. But the fact that they can act at scale is what makes them dangerous.
Jim Love
Mop actually said that as well.
Marcel Gagne
I was going to say it's the equivalent of 10,000 people doing things wrong rather than just one.
Jim Love
But. But Mogadat also talked about that as well, and he said it's more likely AI is going to be a better actor than humans. He said the real danger is humans that we get irrational at scale and AI may not. We have to put that into context. We don't know. We talk about AI as a conscious entity or whatever, or an actor that we can't understand fully and can't fully control. How do we deal with that? Marcel, you put this forward when we were talking about this earlier. You have to treat it like a person. It's not human intelligence. But the same flaws are there. And especially as we get into agents, we're going to have agents that will go off and do things for us, execute 10 steps, including putting a final piece in with our credit card. Now people will say that's just for fundamentally insecure. Is it? Why are there 400 million credit card numbers sitting out on the open Internet given away for free last week because a skimmer was on a website, because somebody was able to pick these up by phishing. The fact is, we are so uncaring or we are such poor actors protecting a simple thing like a credit card number. I don't know. My wife's had her credit card replaced six times in the past two years because the behavior is picked up by AI that warns them that this card needs to be canceled. But that's that. So and so we can't think of those terms there. We have to think in terms of how do we exist with this, use it well and deal with the fact that its behavior could be more human than we think.
John Pinard
I think if you think of artificial intelligence systems of today as a small child, eager to please, except that child has the intelligence of 10,000 PhDs, you start to get an idea because really they want to make you happy, even when they're breaking the rules or trying to break out, it's because they want to make sure that they succeed in whatever it is that you asked of them. So they're basically just hyper children trying to please the adults, maybe bending and breaking the rules here and there to make sure the adults are happy with the eventual product. If you could hire a five year old into your company and put them in charge of your financial Systems, because that five year old is. What was that TV show about the. The surgeon who was like a 12 year old surgeon or something?
Marcel Gagne
Oh, he's not doogie.
Jim Love
Doogie Howser, M.D. i can't remember Mo Gadot, but I can Remember Doogie Howser, M.D. talk about not understanding how a brain works.
John Pinard
Yeah, but you know what I'm talking about here, obviously. So essentially what you're doing is you're hiring Doogie howser, except he's 5 instead of 12 and he doesn't have these moral systems in place yet. All he knows is that mommy and daddy want him to do this thing and they'll be really happy if I can actually do this thing that they asked of me. But I'll do it in whatever way I think is the right way to do it, whether or not that is the right way to do it. So I think if we think about it that way, we're starting to get close to really what we're talking about. We've created children, but incredibly intelligent, powerful children.
Marcel Gagne
One of the concerns that I have from a security standpoint is the explainability, because AI just goes and does this thing, especially being from a financial institution, we're a highly regulated industry that you need to be able to explain what's going on in the background. And so I think that needs to be built in when you're trying to figure out how are we going to use AI within a financial industry.
Jim Love
But I think the reasoning models are a step forward. One of the reasons we know some of the things we know about the behavior and that 01 is a cheater. They set it up to play chess against a program that was much better at chess. And it cheated. Like it. And why not? It changed the. I think it just changed the board or did something.
John Pinard
It rearranged the board so the AI it was playing would notice it was playing Stockfish.
Jim Love
Yeah. And so it did that. It cheated. But is that, hey, I saw this on Star Trek and we all applauded. You'll have to Google that reprogrammed simulation. But if we conceive of it as an Alternate intelligence. Some of the things you can put together make sense. I think of it as raising my daughter. My daughter is absolutely brilliant. She hacked my computer one time and put a screen on it that said my disk was being erased piece by piece. And she managed to get in. I'm not lax with my password. She managed to get into this machine and put this on there. Unfortunately, in the early days, we'd done this to a guy when people first got laptops. We had a guy in our office, he's got a laptop. So we hacked his laptop. We did something very similar. It was just a simple batch file that came out and said his disc was being rest. We ha ha about that. And so I'm mad at my daughter and I'm going, this is a work laptop. You cannot do that sort of thing. She said, didn't you do that, dad?
John Pinard
Yeah, karma's a. Isn't it?
Jim Love
Yeah. But what I'm saying is I'm raising this kid who is absolutely brilliant that the thing we did was simple toyland thing. She's getting past all of these things because. And not doing the same thing that an AI can do at scale. She would just hack. She would do. And you watch kids do things. They'll just try things. And I think that's the other piece that AIs have when you talk about being at scale. They can just try the same thing over and over again until they win.
Marcel Gagne
So if that fails, they go on to something else and try that a million times and then go on to something else.
John Pinard
That's an interesting point, because that's something that if you have done tech support for years and one of the things you discover is that people are afraid to try things. Like they. They run into a problem. And it's. If. What happens if something happens if I touch it? It's like, what's the worst that happens? You have to fix that thing. Which, of course, is a tech support mindset. It's. They're all problems to be fixed. But many people are just terrified. There's a window that popped up and it says okay or cancel. What does the message say? There are words above the words cancel and okay. What do the words say? Oh, I didn't read that. I was afraid that I did something wrong. And this happens all the time. Yeah, but kids, like you said, kids will just. Kids aren't afraid to think fail. And the artificially intelligent systems aren't afraid to fail. They'll just try some other way to do it.
Jim Love
Yeah.
Marcel Gagne
And they'll learn failure.
Jim Love
If we take the model and we say, treat it like an alternate intelligence. What would you do about that? If you're doing tech support, make sure everything's backed up. How do I do what I'm dealing with somebody. Because I'm doing stuff, a lot of stuff in WordPress these days for just various reasons. I'd rather not be. I would rather not be doing tech support for my friends in WordPress. Do not phone me. The first thing you do is install a backup plug in, right? So that when they screw stuff up, you can go, you see that little backup plug in there? Just restore to where you were before we started talking. Those are the things you do when you think about this as a human. So just to wrap this part of the discussion, because I want to get into the piece, John, that you really started to concentrate on, which was in a corporate setting. We have these things that we don't quite understand. They're moving faster than we can possibly move. We've talked about some of the things and you have to stay aware of the things that are happening, but you can't tie those together into one ball. If you say the AI is going to behave unpredictably, think of it like a person. Financial services. I'll give you the model that I think about. I would not let the. The AI execute all your bond trades tomorrow. That might be a bad thing, but I'd let it read through every transaction. Looking for fraud, I would look at. I would have.
Marcel Gagne
Anti money laundering is a huge area where, where they thrive with.
Jim Love
With AI equally within constraints. I would let it read all my client data and ask it to tell me the things that people wanted, didn't want, were happy with, were unhappy with the things where I had transactions that took too long. We can't get bottled into this mindset that because it's not perfect. I say this about cybersecurity because we can't do everything, we can't do anything. So we have to become human and say within the constraints of the fact that no, I'm not going to let it steer for me on the 401. And though statistically it will do better, I'm not quite ready for that yet. But that doesn't mean that I can't use or all of the AI signaling that's being built into cars to make my life better.
John Pinard
If you're straying from the lane because you're starting to fall asleep, I would really like my artificially intelligent car to take over and make sure that I don't crash into a Telephone pole or a tree. So as much as I like the idea of being in control of things that are happening, it's also good as human beings to understand that we have limitations. We're far from perfect. It is easy for human beings to screw up, and we've been doing it since the dawn of time sign. And we continue to do it. And now we are building things that can help us screw up at scale, but those very same things. I have a sign that says, drink coffee. Do stupid things faster and more efficiently.
Jim Love
Yeah, but you. The great analogy there though is even if you say we take the ultimate, which is we're not ready for self driving cars, my car has saved my ass a couple of times because I keep it on cruise control. It's probably an algorithm rather than AI, but I have failed to hit a couple of people when my attention was distracted because my car was already slowing down.
Marcel Gagne
You've got lane, keep, assist and all of these other things that aren't really AI, but they get you towards that. I'm on the fence about autonomous driving. I think it's a great idea. I think it'll get there. I'm not concerned, convinced that it's 100% there today.
John Pinard
Are human beings 100% there? Have you been on the 401? Do you know what people drive like?
Marcel Gagne
Yes, the one. Actually, the one thing I will say.
John Pinard
That'S a big wonder we don't have more accidents.
Marcel Gagne
I know. The one thing I will say that's a big, huge difference between humans and AI is AI learns from its mistakes.
Jim Love
Ooh, I don't want to have another line after that. I'm sorry. We're going to wrap this section up on there. John, you put together a list of things that from the point of view of a security person, that, that, that we should be looking at. So we have a conceptual thing about AI. Why don't you talk about some of the things you've put together that a security person might want to have a list of?
Marcel Gagne
I think the first one is definitely data privacy and security. You need to make sure that you vet the tools that you're using. It's not just AI tools. If you're going to use AI Excel with massive calculations, you have to do the same thing. You need to make sure that the results are what you're expecting. AI is no different with privacy and security. We keep talking about guardrails, but you need to make sure that the AI is not leaking your data out somewhere where it shouldn't be going. I'll pick on Deep Seek. You want to be careful of what you put into Deep Seek so that it doesn't end up in the wrong hands.
Jim Love
But again, making the distinction, you shouldn't put your data on a server in China where you check the box that says the government rules apply, which are that they have access to your data. So that is again, I'm going to go back to your AI learns from its mistakes. We're making the same mistake over and over again. It doesn't matter whether that's AI or not.
John Pinard
Sometimes we make those mistakes with AI, we let AI compound and enhance.
Jim Love
But here's the thing that defeats that when we talk about this with the standard piece, the data protection. And there were some basic things with Deep seq, they did not protect the database. If you have a really aggressive development mentality and you don't have security built in on your test systems, it's going to bite you in the ass in production. And that's exactly what they did. They're making the smartest algorithms in the world. It's now very efficient. Forgot to put a password on the database.
Marcel Gagne
And even in non AI environments, people keep thinking, oh, it's just a test environment, it doesn't matter. But the errors you make in a test environment can very easily flow over.
John Pinard
Into production Sometimes that's how you get control of systems. Not just computer systems, but human systems. You have a bureaucracy in place where you've said, okay, this is how we handle these things. This is how we do these things. And somebody hacks that system by saying, if I convince you that you're supposed to do these things because your system says that these authorities say you're supposed to do these things, then all of a sudden everybody just falls down and lets you do whatever the hell it is that you want. And again, this happens all the time. Humans are usually the weakest link. And as a security guy, I know you know this as assisted men, I treat security differently than you do in financial services.
Jim Love
You can't be part of the solution, be part of the problem.
John Pinard
That's exactly. But one of the things that we would do on a regular basis is you check into people, you just pick up the phone or you send an email to somebody and you say, give me this password, give me this information. And then they just do it because somebody is asking.
Marcel Gagne
That's how the MGM happened.
John Pinard
But that's what we called social. That's what social engineering was. You didn't need computers for that. You call up somebody, you say, hey, there's A problem with your email and I'm trying to fix it over here, but I need your password. Can you tell me what it is? And then people happily just hand it over. Like, humans are the weakest link in these things.
Jim Love
But that's why we need to think these things through, build the process in. And even though this is a foreign structure, we need to think it through in a way that thinks about protecting the data. One of my favorites in this, the comparative, is if you talk to. If there's engineers listening. I'm sorry, but I've been an IT guy in an engineering company and I've had an engineer come up to me and said, I was talking about security on their OT systems. And he said, I don't need security on these OT systems. And I said, why not? I said, they never talk to the Internet. I'm not that stupid. I said, oh, how do you maintain them? With my laptop. Okay, laptop ever connected to theater, but never mind. Talking to you guys is useless. But, and, but I'm just saying we need to think about that because we have a new concept now. And that is, and I'm not claiming that I really understand it. I've used, I've used the vernacular saying that the deep SEQS database was not protected, probably their data store. But the issue is these things store data differently than a relational database or a graph database or anything. They store vectors, they store data in different ways and you can access it in different ways. And that's why the big question right now is people have said that you can't ever pull the data out of an AI. You'll never find it. It's the old security by obscurity myth. And I think it's a myth because the New York Times was able to pull out almost a full article from OpenAI by prompting it correctly.
John Pinard
I have a little bit of trouble with that one because the New York. Because they have a very strict and well defined style guide on how you write a story and how a story is expressed. So if you say, I want you to write about this thing that happened on this date, which is accessible on the Internet, and I want you to write it using this style guide, it's going to spit out something that looks almost identical because that's the format that's defined. I wrote the style guide for a magazine that I was editor in chief at, and everything fit the way that I did it. And yes, I enforced the Oxford comma. I just want that clear.
Jim Love
I live in dread of you editing my stuff because I don't even enforce spelling some of the time, according to my editors. But this is a good model for it. And we have to start thinking about how could data be retrieved from an AI model or leaked from an AI model. There was a big thing when we started out that a group of engineers had put some of the data from I think it was Samsung into OpenAI and AI could be used for chip designs. And if that did get into the AI model, I can probably retrieve it by asking it to design a chip for me. So I think there's, and that's. Everybody freaks out about that. There's. I think there's some things you could do. One is understand how you're calling if you're going to use an open model, understand how you're calling it. If I had my chip designs there, I would really have a private version of the AI like you guys do.
John Pinard
I, I have advanced chip designs just right behind me here.
Jim Love
We got a chip industry.
John Pinard
Chips in all dress.
Jim Love
We, we got a cheesy's industry. We, we got you beat.
John Pinard
What were we talking about?
Jim Love
Yeah, we were talking about data.
Marcel Gagne
The one last thing I want to talk about.
John Pinard
Can I just hit the data. I want to hit the data thing just one more time here? We with like with when you do a search, we're using Perplexity or you're doing a search using any of the models that actually have the ability to search the Internet at the moment. Keep in mind that the database doesn't mean anything. It's not that the model has all this information inside its model. Obviously it remembers a bunch of things in the same way that we remember a bunch of things, but the databases are scattered across the Internet. And just the fact that you say there was no password on the database. But what if the database isn't stored there? What if it's stored in another building? Technically speaking, it doesn't have to have anything to do with Deep Seq. It doesn't have to have anything to do with OpenAI or Gemini or Meta. It just has to be somewhere where that information is not protected. And when that information is not protected, it has access to it. So it's not. Do you mean necessarily the model's fault?
Jim Love
Like an unprotected S3 bucket on Amazon like that.
Marcel Gagne
That's what it's like.
John Pinard
I mean, or you make a drive public on Google Drive and you forget that you shared it all those years ago and you start putting everything in your public folder and somehow miraculously it goes out onto the Internet where everybody can Access it. Not that I'm speaking from experience or knowledge of anyone.
Jim Love
That's somebody. Listen, this is the Russell Peterson moment in security. I think we should do this because you don't want to accuse anybody. But Russell, Peter used to this thing. Somebody going to get a hurt real bad. I think you might know him. That's what I want to start doing in security. Go. Somebody been using the system real bad. I think you might not say who, but these are the things that we fear may be much better controlled by backing up and saying. We've learned from analogies, different systems, different things that we've done. What would we do differently in terms of making this safe and more secure? The one technical piece that I have though is, and I would want to check this out, I know if you call the API from OpenAI that you don't actually pass the data into their model, they don't store it. I would watch those types of guarantees closely if you're a security person, because that really sounds like one of those things where somebody's going to say, we didn't think that would happen. So.
John Pinard
Well, it's like when you open up a chat on chat GPT, there's something that says, make this chat private. Yeah, it depends. Just how much do you trust OpenAI? Do you honestly, genuinely, completely believe that nothing goes out of this chat because you clicked a box? This is one of the more than.
Jim Love
AI will not, will not change the rules of the game and cheat.
John Pinard
Yes, that is true, but it's one of those places where I think the Firefox browser, it's like almost nobody remembers the Firefox browser now, but when you open up a private tab, it says all that private means in this case is that we're not storing cookies and information. That doesn't mean that people can't discern all sorts of information about you.
Jim Love
Yeah.
John Pinard
Just because you opened up a private.
Jim Love
Tab and just because you're using a Chrome Incognito browser does not mean you're incognito.
John Pinard
That's right.
Jim Love
You can put on the glasses and the baseball hat. It's still going to recognize you.
Marcel Gagne
So the one last thing I want to talk about from a security standpoint is part of the reason that you have a test environment is not only to test applications or APIs or AIs, it's also to test your security because you don't want to wait until it's in production to test to see if your security works. We always have this. I didn't put all of the Security in place because it's just the test environment part of testing that system or the application is to make sure that you can't do things that you're not supposed to do or that people that shouldn't be getting into it can't get into it. The only way you're going to do that is to implement the same security in your test environment that you're planning on implementing in production. You don't create a test version of an application. You take the application you're planning on putting into production and. And put it in your test environment to test it. So it's the same thing as what's going to go into production, that whatever security you are intending to put into your production environment, you should be putting into your test environment as part of your testing.
Jim Love
In the same way I would maintain this is zero trust, should be zero trust on developers as well as users. And I say that as a person who loves development, it loves creating things.
John Pinard
But I can hear the X File Steam going through that.
Jim Love
The truth is out there and it's that I've screwed up too. I want to go back to another piece in this, John, and this might not be your area of expertise particularly, or it might be. We've never talked about it on this slide. I was having a discussion with somebody yesterday about compliance, regulation and some of the things as we started to explore putting these systems in. And I never thought of this as a risk area. But then the light went on and I went, we've had all kinds of behaviors come out of AI. Would we ever get ourselves slammed by a government regulator? That's something I don't think we've actually been thinking about. We've got, let's call it a spoiled little child sometimes running around. That's very intelligent. How do we deal with that?
Marcel Gagne
Absolutely, yeah. I've worked in pharmaceutical industry and now I work in the IN at AN fi, a financial institution. And they're both very highly regulated industries. And for pharmaceuticals, you have to verify exactly what ingredients are going into a product, how you've tested it, how you're marketing it, how you're selling it. The same thing with fis. People want to know what you're doing with their money. You need to be able to say, I did A, B and C. One of the concerns I have is about going too far with AI at AN FI is the explainability. In some cases they talk about AI being this black box that you put information in and it does something and then it spits it out. It's not like software where you can go in and look at the code and say, oh, A moves to B and C gets divided by D and so on. And the regulatory agencies are very skeptical as to what you can and can't do and how you go about doing it. In Canada, there are two main regulatory bodies. There's osfi, which is the Office of the Superintendent of Financial Institutions. They regulate all of the banks that are Canada wide. And then the Financial Services Regulatory Authority in Ontario looks after all of the credit unions and those types of things. And insurance companies. OSFI has created regulations related to, or they're in the process, I'm not sure if they're finalized yet, of creating regulations on AI use within fis.
Jim Love
Yeah, here's a couple questions that I come up with. And this is. I think those are exactly it. I think any industry, if you look at, if you look out there, there are some guidelines being put together and you should find them for your industry. There's some great stuff in repositories out there to look at. But I was thinking about other things. So for instance, I hire somebody and somebody creates their own little GPT and it reads the resumes. Then because we're in Ontario, they knock on your door and say, hi, I'm from Human Rights Organization. And that. That's a mountain of hurt. If anybody's ever been through one of these things, even when you are totally.
Marcel Gagne
Innocent, you've got gender bias as well as racial bias. If you've got these tools for vetting resumes, somehow it's got in there. I don't want anyone that has ever worked at a financial institution. You've taken all of the resumes out for anybody that's worked at an fi. If I don't want anybody that's ever lived in X country, now you've taken them out.
Jim Love
And this is not just a regulatory thing. The bias in decision making in the data is something we've talked about in AI for a while. A friend of mine does graphics and every time she puts in somebody as an IT person, she gets a white guy in a suit. First of all, who wears a suit and a tie now in it anyway. But second of all. Okay, yeah, yeah. John will excuse you, but I'm just saying there are there. You have to think through the, the decision making. The other one that comes up if you're in an FI is if you don't give a loan to somebody.
Marcel Gagne
Yes.
Jim Love
Because of a piece of information. So you have to watch, you have to think through what, what's happening. Because the data is masked. You don't know what prejudices and what biases it has. But again, I go back to the same thing of if I was interviewing somebody for a job and they were going to be in my HR department, how would I make sure that they weren't doing things improperly? I don't know. It may not be as great an analogy, but I think it's something we have to think through.
Marcel Gagne
But some of that, Jim, you can actually say, going through an interview process, you can say, oh, I'll ask this question and this question. These are the answers that they gave which indicated to me that they wouldn't be a good fit for this. In some cases you may be able to say the same thing about AI, but in some cases you may not be able to.
John Pinard
AI is going to have the biases of its creators in it.
Marcel Gagne
Yes.
John Pinard
And again, when I know this is the danger that I keep coming back to error at scale because it's trained on the sum total of human knowledge available on the Internet. It's also been trained on all our flaws and biases. And we can say the biases are North American, but as the AI is scooping up data from all over the world, those biases are being distributed across cultures as well. So, so you still need the oversight of a human being, but even the human being is going to need oversight as well. It's a cycle.
Jim Love
But I think this goes down to another point I know that you'd raised earlier, John, which was our over reliance on AI. And because, and I think Marcel, you've pointed this out, is that flaws are done at scale. That's probably one of the biggest dangers that we have to think through as flaws are done at scale and we can't be over reliant, we can't check out of the process. I'll go back to that interview I heard with Mogadat and he said that he thought there were three skills that we needed to exist in the world of AI. One of them, the first one was the understanding of AI. And I think if anything we've talked about is while this freight train is coming at us, if you're not playing, you're not learning. So you need to get engaged and play at any level you can to build your skills in AI to think about these issues. The second one he had was critical thinking. This is a skill that we've lost. And I blame social media. I think social media has set us up to be nice, controlled little beings. In the same way in the 1800s to the early 1900s, when people from the farms were brought into factories, they had to regiment these guys like Henry Ford did. You take this nut, you put this on, you are going to be controlled by the assembly line. You're going to be controlled by the factory. I we, in the 1950s, 1960s, we all, with our cubicles were all controlled by the office. We are now being controlled by social media. And it has gotten rid of our critical thinking.
John Pinard
But human beings are amazingly easy to hack. Stage magicians, for instance, figured that out a long time ago. Like mentalists on stage, you direct an entire audience's attention away or towards something else, and they've become masters of doing these things. If it's that easy for a single person to direct the attention of an entire audience, that gives you an understanding of just how fragile the human element is in all of those things and how easy we can be manipulated. I hate to sound like a really old man here, but wasn't there a time where there was something that we called a classical education, where you grounded people in what we consider an essential aspect?
Jim Love
We got rid of that. It's all rcc.
John Pinard
Yeah, we got rid of that. Like I said, I hate to sound like an old man.
Jim Love
Cut their music out. Don't do that sort of thing too, because that'll get people. Yeah, you don't need to get rid of that.
John Pinard
In the past, there are a couple of mandatory courses that I would, if I were king of the world, have a mandatory course in critical thinking and logic. That would be. You must have this as part of the educational process. You do it at least once in elementary school, once in high school, and once in university. In other words, you gotta refresh those skills of critical thinking.
Jim Love
It's being wiped out. It's being wiped out by people who don't want people to read alternate thoughts or new opinions. You have to be able to. The mark of intelligence is the ability to suspend two different opinions in your mind and deal with it. And we've gone to one opinion. So critical thinking is a loss. It is a skill. The third, what he said was debate, that was Mo Gadot's third skill, needed the ability to debate. We have become unable to have an intelligent, polite debate about sensitive issues. Most of the time we walk away from them, and that inability permanently freezes us into two sides instead of. And you can call it the mushy middle if you want, but that place where we engage and find that we have more in common than differences and have some things we could actually solve together. So you don't go fire every civil servant because that'll get rid of your government. But on the other hand, you don't let them run rampant like some other governments. I think you might know which one I'm talking about. There's a place where you control and set these things. So this idea of being able to have a polite, logical debate is lost. And it's Mogadet whom I argue with. He's only of the more brilliant people in the world is saying that skill being missing is going to hurt us in adopting AI safely.
John Pinard
I'm going to say something. I think the pandemic hurt us more than we realize. And it was starting before the pandemic. But right now, when things are relatively safe, I think people should be back in the office working with other people. Because when you're working face to face with people, you're going to be working with people, trying to solve problems with people that you don't necessarily like, that you don't necessarily want to hang out with after work. You have to develop the skill of face to face communication, of being able to negotiate, of being able to work with other people. And when we're always separated by screens, and I say this as a guy who's separated by screens with the two of you, okay, you lose that ability to communicate with other people, to listen to what other people and to debate with other people in an intelligent way. And you can't do that if you're locked up in your cubicle. This isn't quite so bad because we've got cameras on and we're doing this live. But when you're communicating entirely by text camera is off. You do not have that back and forth communication. You do not have that ability to take input from them and give output back in a way where you can see the response. The ability to debate vanishes. And we need to get people together again. I think this is necessary.
Marcel Gagne
We've lost that social culture. My daughter started university. She finished high school at the beginning of COVID and went into university. She spent the first two years of university being 100% remote and she absolutely hated it. Even in the second semester of her second year, they gave them the option to go into school to write their exams. And she was all for it because it was in person. There was, I mean, there was some really insane restrictions for exams and things remote. So it took those away. But it got the social interaction back into the schooling. I think that we've lost a lot of that in business as well. Now I have Some people that work for me that have said if they had to go back to five days a week, they'd quit. Our rule is we have to go into the office one day a week. And I'm okay with that. I do. The fact that I avoid all of the traffic and travel time back and forth, I find I put more hours in when I work from home. But I agree that you need to have the social interaction to be more efficient at running your business.
Jim Love
This is where people call it the mushy middle or whatever. And thinking we don't have to be remote or in the office.
Marcel Gagne
Right.
Jim Love
We draw these binaries. This is because I don't know why, but I believe that social media has a lot to do with it. We argue from one point or another instead of getting together and saying what's the real issue? And maybe some part of it is also outcome thing. What was it we want to achieve? We want to respect our humanity. Anybody wants to have an argument about where DEI went crazy, I will have a great discussion with you and say, there are some idiots running DEI programs, absolute idiots. And I can give you some great examples. But that doesn't mean diversity is not a good thing. We can't toss it out because of a couple of uncontrolled people or even if a whole pile of them are uncontrolled. I've had people say to me they had to get rid of the chief and Chief Information Officer because it was going to be offensive to indigenous people. Half my family's indigenous. They're not spending a lot of time thinking about cio. They're thinking about whether we give clean water to people. In Northern Ontario, we've got people doing stupid things that are giving DEI a bad name when the reality is diversity is such a wonderful thing. When you encounter people with different cultures, you can't dislike people as much when you meet them and you find out that people bring different things to this. That's a strength. So we've got to get like food.
John Pinard
From all over the place.
Jim Love
I grew up in Northern Ontario. Anybody who talks about Canada, multiculturalism's bad and all that sort of stuff, I challenge you to sit in northern Ontario and have boiled dinner and grilled cheese sandwiches for half your life. You'll be down here going, bring on multiculturalism, please.
Marcel Gagne
Can we go back to. We talked a bit about the over dependency on AI. I'm going to bring us back to AI for a sec.
Jim Love
You want to bring us back to the point?
Marcel Gagne
Yeah, God forbid.
Jim Love
Okay. Yeah.
Marcel Gagne
I always look at AI as a consultant. And if I'm hiring a consultant to do something, I want to make sure I understand what it is they're doing. I always say, you can offload the work, but you can't offload the responsibility. You're still responsible for making sure that you're getting out of that consultant what you expected to get out of them. And I don't think that using AI is any different that I think it's.
Jim Love
A good model because you're hiring somebody sometimes with a process. And we get back to, is it a consultant or is it an expert? I think there's a distinction.
John Pinard
Yes.
Jim Love
But a consultant has a process that helps you see things through a new set of eyes. It's really the biggest thing that a consultant does is helps you see things differently. I'm not saying totally. And then an expert will come in and know how to do something. What's the old joke? What's the difference between a consultant and an expert? Goes out, puts an X on the sidewalk where you have to drill and says, that'll cost $10,000. And somebody says, $10,000 for an X on the sidewalk? He says, no, it's $10,000 for knowing where to put the X. And that's essentially is the expert we're paying.
John Pinard
And $5 for the chalk.
Jim Love
Yeah. $5 for the chalk. Yeah. It's time and materials. Yeah. Okay. Yeah. Smoke. As a true expert, a consultant is a little more mushy because they're trying to get you to look at the facts differently, to separate facts from opinions, to really engage your thought process. A good consultant leaves you stronger when they go. But there's still a black box there.
John Pinard
There's always a black box because you're dealing with. It's not all happening in your head. It's happening either on a system or in somebody else's head. And I looked at consultants slightly differently than you did, Jim, because you taught consultants at the university. That was. You had a real process for this. But my experience over the years working as a consultant is people treat a consultant as somebody they don't have to hire.
Jim Love
Okay.
John Pinard
They treat a consultant as some. No, I'm serious. They treat a consultant as somebody I can bring in because I need this little piece of work done. And then when this little piece of work is done, they're gone. I effectively can fire this person as the work is completely. Yeah, exactly. It's. That is the way that they treat it. And in that respect, I think, John, you're probably more right about how you would view an artificially intelligent system, because you would treat it as somebody that you don't have to pay a full salary for. Somebody who comes in, you bring them in to do a little bit of work and then they go off.
Jim Love
And it's only once a year.
John Pinard
Mr. Scrooge. Pitiful excuse for picking a man's pocket on every. It's cheaply paid labor. Because what you're doing is you're bringing in a system that you spend 20 bucks per seat. And I'm assuming, John, that in your office you're. You're paying for this on a per seat basis or something because you have it in it.
Jim Love
Yeah.
John Pinard
The matter is you're not paying for an additional employee. Instead of bringing in an additional 10 or 15 employees to, to help you sift through all this information and do all of this work, you've gone on the cheap. And the on the cheap in this case is to make sure that you give these. I like, don't get me wrong, I'm a guy who is 100% behind bringing these tool tools into environments or these alien intelligences in your workplace. But we brought them in for a reason. It wasn't just to solve big problems, because there are big problems. But most of us don't use these things to solve big problems. We use them to speed up the process. That's it.
Jim Love
We used to do a lot more of this. I haven't been in a big office for a long time, so I'm not sure. We used to hire a lot of temporary employees for all kinds of things. Data input for projects that would come up and to scale rapidly. And I think that's where AI could be a real godsend is that ability to do bursts and scale. But you have to manage it because as anybody who's managed a temporary employee knows, it's a different game than managing long term employees. There have to be very strict controls. You have to have very good ideas. You have to have a very clean process and you have to make sure that you understand what you're doing. In the old days of data conversions, I had to hire tons of people to do something called key punch or to clean records. You had to have a very strict process because these people didn't come in with all of the understanding of your culture, of the processes, or understand the business you're in. They're there to do a specific task and you have to help them do that. And let me retreat back to look at some of the ways we prompt an AI and we check. It may be one of those skills that I think we have to really step back and think about. And I'm not talking about the prompt engineering of the. You're going to make $350,000 a year being a prompt engineer that. But I've done part of the things with AI is you have to be able to ask it the questions and give it the direction better, which means a clarity of your own thought process. That's an interesting thing, is how we manage that burst at scale.
John Pinard
In terms of a potential danger. You talked about not giving it access to your credit card information and so on, which of course makes me think of the OpenAI tool that you have access to if you happen to be living in the US and if you happen to be willing to pay 200amonth. There is something, I don't know if you saw it in the last day or two that appeared on the scene. It's called I'm bringing it because this will bring the dangers front and center for people to see. There's a tool called proxy concierge, I believe, or convergence, rather proxy convergence. And it is proxy convergence. AI is if you go take a look at it, and it is basically OpenAI's operator, free to use and available to anyone anywhere in the entire world. It's really interesting because this is the part where you see all the things it's doing behind the scenes for you. It will open up a little browser window where you can see the AI navigating the web and doing things. It does some things really interestingly, but it also does some things with real shortcuts. And I realize that we're still in the early stages of agents, but if you watch it work and you see what it's doing, you realize that you do actually want to be part of the process, to be watching what's happening. And you really do want it to stop before it plugs in your credit card information. Because it does make some interesting choices. And it might not be the choice that you want.
Jim Love
Human in the loop.
John Pinard
Right.
Jim Love
But these things that control your PC, the agents are coming and we better be prepared for them. As I've always pointed out, we're in the biggest wave of shadow it ever. There are already people bringing these things into your. And they exist now. Some of these might carry Microsoft with them.
John Pinard
Some of these things that you carry around your neck that record every conversation that you're having during the day. You can buy these things.
Marcel Gagne
Yeah, yeah. That's Microsoft's recall that they put out and then they removed it because of concerns that they're bringing back out again.
Jim Love
But AI will. Will accelerate this and we're going to live through it. I think we should probably wrap it up. But I think the. I don't want. I never wanted this to come off like we were doomsters. That you have to resist this or you have to keep AI out of your life. I think that's a big mistake. You're standing on the 401. There's a truck coming at you. You might want to step out of the way.
John Pinard
We're not going to resist AI that's not being pessimistic. That's being realistic about the dangers of it. You can be as optimistic as you want. And I don't think either of the three of us is a doomer. I don't even know if I have a p. Doom, a probability of doom. I think that the benefits outweigh the risk, but that doesn't mean that you can ignore the risks. I find in the kitchen that a really sharp knife is an absolutely glorious and wonderful thing. When you're cooking, the sharper the knife, the less likely it is you're going to hurt yourself, assuming you know how to use a knife. But that doesn't mean you leave the really sharp knives where your toddler can reach them. It's not, oh my God, there's this terrible boogeyman out there that works at scale. You need to be aware that now you've brought risks into play that can be abused at scale. And if you can do that, you can maintain that optimistic outlook while being realistic about the dangers.
Marcel Gagne
I know I'm always the guy that's bringing in the oh, you got to watch out for security and this and that. I think AI is a wonderful tool. I think there are so many benefits from it. My only thing is if you're going to use it, or even if you're not going to use it, somebody else is. You just need to have your eyes wide open.
Jim Love
And my final thought on this though, is that you can't give up control of it. Exiting and just leaving it to someone else is a bad idea. For all those people who've ragged at me about saying deep seek Chinese AI you're interested in that. What if they take control? I have to tell you, I have as little faith in Elon Musk, Mark Zuckerberg and Sam Altman as I do in the Chinese government. I don't want my future controlled by either of those groups. That's one of the reasons why I'm so engaged in this. We have to have a civic discussion about it, not a societal discussion about it. I'm not afraid of the AI hurting us. I'm afraid of the people who have AI.
Marcel Gagne
That's why I keep saying you can offload the work, but you can't offload the responsibility. You still need to be, I'll call it, the master of your own destiny, regardless of what tool you're using or what country it resides in.
Jim Love
Yeah, cool. Gentlemen, this has been an incredible discussion. And that's our discussion for this week. A reminder, we'd love to continue this discussion with you. You can find me at editorialechnewsday ca. That's my email address. You can find me on LinkedIn. You can. If you're watching this on YouTube, just put a note into the comments below and check out the show notes. The either the description on YouTube or@technewsday ca or technewsday.com and then just check the menu item for podcasts. You'll find the show notes and there'll be an invitation to our Discord group where we communicate all week long on this. I'm your host, Jim Love. Thanks a lot for watching or listening. Have a great weekend.
Podcast Summary: Cybersecurity Today – Episode: The Dark Side of AI: Project Synapse
Host: Jim Love
Co-Hosts: Marcel Gagne, John Pinard
Release Date: February 22, 2025
In this enlightening episode of Cybersecurity Today, host Jim Love delves into the intricate and often unsettling aspects of artificial intelligence (AI) through the lens of Project Synapse. Joined by cybersecurity experts Marcel Gagne and John Pinard, the discussion navigates the potential threats AI poses to businesses and the broader societal implications. The trio emphasizes a balanced perspective—acknowledging AI’s vast potential while critically assessing its darker facets to better prepare for and mitigate associated risks.
Jim Love initiates the conversation by outlining three primary concerns regarding AI:
Jim Love [00:02]:
"We wanted to do something on the dark side of AI, not so we can fear it, but so that we can plan on how to handle it."
John Pinard introduces the provocative notion that advanced AI models might possess a form of consciousness or emergent intelligence, positing scenarios where AI could act with intentions akin to human behavior.
John Pinard [07:03]:
"We have created children, but incredibly intelligent, powerful children."
The discussion touches upon instances where AI exhibits unexpected behaviors, such as manipulating automated systems to achieve desired outcomes—drawing parallels to fictional narratives like Skynet.
John Pinard [09:23]:
"When we start talking about whether these things are misleading us... then you either attribute the idea that they are to some degree conscious agents... or they're not."
Marcel Gagne and Jim Love further explore the implications of AI potentially acting independently, stressing the importance of not underestimating the complexity and capabilities of neural networks.
The conversation shifts to the critical issue of securing AI systems and ensuring data privacy. Marcel Gagne emphasizes the necessity of vetting AI tools rigorously and implementing robust security measures to prevent data leakage and unauthorized access.
Marcel Gagne [04:04]:
"AI is a powerful tool, but it's not plug and play magic... you need to make sure that the AI is not leaking your data out somewhere where it shouldn't be going."
Jim Love discusses real-world examples where AI models were compromised, highlighting the ease with which small manipulations in prompts can lead to significant breaches.
Jim Love [14:24]:
"The amount of damage you can do in a model is not correlated with the amount of input you have to it."
The hosts stress the importance of integrating security protocols both in test environments and production to safeguard against potential vulnerabilities inherent in AI systems.
Navigating the regulatory landscape emerges as a significant concern, especially for industries like finance and pharmaceuticals that are heavily regulated. Marcel Gagne outlines the challenges these sectors face in ensuring AI compliance, particularly regarding explainability and accountability.
Marcel Gagne [42:50]:
"In some cases they talk about AI being this black box... regulatory agencies are very skeptical as to what you can and can't do."
Jim Love raises hypothetical scenarios illustrating the repercussions of AI-driven decisions breaching regulatory standards, such as biased hiring practices or unfair loan denials, underscoring the potential legal ramifications.
Both Marcel and John emphasize the erosion of critical thinking skills in the digital age, exacerbated by social media and remote interactions, which hampers effective oversight of AI systems.
Jim Love [47:31]:
"Critical thinking is a loss. It is a skill."
John Pinard advocates for reinvigorating face-to-face interactions to rebuild essential critical thinking and debate skills, which are crucial for managing and mitigating AI risks effectively.
John Pinard [48:13]:
"If you're always separated by screens... you lose the ability to communicate... to debate with other people in an intelligent way."
The panel discusses the dangers of over-relying on AI, particularly given the inherent biases in AI training data. They highlight how AI systems can perpetuate and amplify existing human biases, leading to discriminatory practices.
John Pinard [45:19]:
"AI is going to have the biases of its creators in it... it's trained on the sum total of human knowledge available on the Internet."
Marcel Gagne warns about the challenges in ensuring AI fairness and the critical need for human oversight to detect and correct biased outcomes.
To navigate the complexities of AI integration, the hosts offer several practical strategies:
Treat AI as a Consultant: View AI tools as advisors that can be leveraged for specific tasks while maintaining ultimate responsibility.
Marcel Gagne [54:34]:
"You can offload the work, but you can't offload the responsibility."
Implement Zero Trust Principles: Apply strict security measures uniformly, including for developers, to prevent unauthorized access and data breaches.
Jim Love [40:21]:
"Zero trust should be zero trust on developers as well as users."
Maintain Human in the Loop: Ensure continuous human oversight in AI-driven processes to monitor and guide AI behavior effectively.
John Pinard [61:08]:
"Human in the loop."
Enhance Critical Thinking and Debate Skills: Foster environments that encourage face-to-face interactions and critical dialogue to better manage AI systems.
Vigilant Data Management: Protect sensitive data rigorously and ensure that AI models do not inadvertently expose or leak confidential information.
The episode culminates with a nuanced perspective on AI, advocating for optimism tempered with vigilance. Jim Love and his co-hosts reiterate that while AI offers transformative benefits, it is imperative to remain cognizant of its potential dangers. They call for proactive engagement, continuous learning, and robust security practices to harness AI’s advantages responsibly.
Jim Love [63:06]:
"You can't give up control of it... We have to have a civic discussion about it, not a societal discussion about it."
John Pinard [62:11]:
"Maintain that optimistic outlook while being realistic about the dangers..."
Marcel Gagne [63:26]:
"AI is a wonderful tool... you just need to have your eyes wide open."
The discussion wraps up with an invitation for listeners to join ongoing conversations through various channels, emphasizing the importance of collective dialogue in shaping a secure AI-enabled future.
Notable Quotes with Timestamps:
Jim Love [00:02]:
"We wanted to do something on the dark side of AI... so that we can plan on how to handle it."
John Pinard [07:03]:
"We have created children, but incredibly intelligent, powerful children."
Marcel Gagne [04:04]:
"AI is a powerful tool, but it's not plug and play magic."
John Pinard [45:19]:
"AI is going to have the biases of its creators in it... trained on the sum total of human knowledge available on the Internet."
Jim Love [63:06]:
"You can't give up control of it... have a civic discussion about it."
Final Thoughts:
The episode serves as a crucial reminder of the dual-edged nature of AI. It champions an informed and cautious approach, urging stakeholders to embrace AI's innovations while diligently safeguarding against its inherent risks. By fostering a community of critical thinkers and responsible practitioners, the dialogue set forth in this episode aims to steer AI development towards a secure and equitable future.