← The Audit Podcast
The Audit Podcast
Ep 238: The Best Advice from Agentic AI Experts
00:21:58
This week on The Audit Podcast, we’ve gathered the best moments from our conversations with Agentic AI experts—featuring standout insights from Charles King, John Thompson, and Andrew Clark. Be sure to follow us on our new...
Loading summary
Transcript18 lines
- [00:00]
A
Foreign and welcome to another episode of the Audit Podcast. I'm your host Trent Russell. And today we have another best of episode. So this one's centered all around agentic AI for the most part as far as the tools that you use. And when vendors start rolling out new tools, then they start talking about agentic AI. It's really not going to affect you that much as the end user. You're probably not even going to notice. But the thing to be aware of especially, and I'm going to give a resource, this is the governance around agents and agentic AI. That's something where audit definitely needs to be involved at some level. And so that's something to keep in mind here. So if you're. If I know things are changing constantly and it's like, what agentic AI, what is this and what do I need to know about it? We're going to try to address that as much as possible on this episode. And then like I said, go to this resource that I'm about to share with you. It's extremely long, you don't need to read the entire thing, but go to that also. Just search around and get an understanding of Agentic. I use whatever LLM you have access to, Copilot, ChatGPT, whatever that may be and just talk to it and help it under help it explain to you what agentic AI is, what the risks are, etc. All right, so the resource I wanted to share if you on our YouTube channel, you can see what I have pulled up this PDF and so you can kind of follow along as I go through this. I'm not going to go through the whole thing. I'm just going to stop the table of contents because it's 63 pages. This is from the Institute for AI Policy and Strategy. This is the AI Agent Governance a field guide. There's also a link in the show notes to this so you can check it out in detail. I'm going to scroll down to the table of contents though, and kind of try to guide you through where I think you should hang out. If you are not familiar with agentic AI, certainly what are AI agents is a good place to start. Page 13. The intro is really good too. It kind of gives you this utopian view of the world if we do things correctly and this just like Terminator style of the world if we don't when it comes to agents and AI in general. So that's interesting and can provide some additional perspective. The risks from AI agents starting on page 27, I think we all need to be pretty aware of. And then there's, from there on, it's mostly around the governance and what's called agent interventions, basically Terminator style. If that happens, we got to be able to shut these things off. That's like the high level, furthest, scariest thing that could happen. And so again, this is a really good resource. It is not easy to go top to bottom on this. I wouldn't expect most people to do so. The language isn't too technical at all. They do a really good job of explaining it, but it is nonetheless, it's pretty dense. So again, if nothing else, pull it up. Go through the areas that you feel like are most important relative to your role in your organization and understand those. And then again, pull up whatever LLM you have access to, where you have questions throughout this and just start asking it questions. You could also upload this to an LLM and kind of say, hey, I'm a, you know, auditor. I'm the cae. From my perspective, what do I need to know? Just use this to some degree, all right? Even if you don't make it through that. But you want to see what AI agents in internal audit actually look like. You want to see the real thing, not some pie in the sky. This is what we could do 15 years down the road or something. Somebody's kind of on the verge of doing this thing. I mean, if you actually want to see it, and especially if you're doing socks work, and especially if you are co sourcing or outsourcing your socks controls testing to someone else, and you want to see how AI agents can do that work for you or them instead, please let me know. You can find me on LinkedIn. Shoot me a message. You can just put the word socks in there. I'll know the context. Or you can email me Trent T R e n t.russell r u s S E L L at G S kanalytics. Com Again, just subject. You can just put the word socks in there. I'll know what that means and I'll make sure that you get to see exactly what I'm talking about. It's very real. There's probably not a better way. You can listen to this entire episode and there's going to be a lot of good context. I can't imagine a better way than to actually see what's happening. For you to go, oh, that's what this is. And here's the potential implications of it. So again, you co source, outsource your stocks work primo type of folks that need to take a look at this. Let me know. We'll get you taken care of. Okay, onto the show. So, first up, we have Charles King. He's the US AI and internal controls leader at kpmg. So pretty solid resource on this topic. And he's just going to kind of give a general overview of agentic AI. If you ever see. First of all, you should connect with Charles on LinkedIn and if you see where he's speaking at an event or doing a virtual thing, absolutely attend that. He's fantastic. Does a really good job of explaining agents the role in internal audit.
- [04:49]
B
In simple terms, what agentic AI does is you can say, I want to write an email to Trent Russell about this audit. I can reference a file and it will actually go. It will open your email application, it will create an email template, it'll draft the language in the email template. It might even go into the global address book and find your address and sort of do everything right up until the point where I hit send. So all of the formatting and all that, I mean, that's a fairly simple agent. You can imagine more complex use cases. But agents fundamentally, unlike knowledge assistants, which a lot of us have, where you can ask questions and get answers and that whatever you do with those answers is on you, agents are actually going and taking action for you. And they're really powerful and really flexible. Because if you think about the solutions we had for doing that kind of thing in the past, things like RPA or even some of, like the powered suite of tools that help you create automation, they were very fit for purpose. You know, you have to train. If I wanted to, you know, have something that would draft an email, you'd have to have a, you know, a kind of workflow just for that. And I would have like an email, you know, tool. But the agentic AI, you don't have to do that so much anymore. If you say I want to send an email to, it can take those words and translate that into all the actions that you need to take. And if you think about, you know, if you think about a lot of the actions that we take, regardless of where you are, whether you're a chief audit executive that's been doing this for 20 years or your brand new, you know, beginning auditor, a lot of the things we do every day on our computers are, I need to go find that file. I need to, you know, where was that table again that I was looking at? Or, you know, whatever it is. And if you can just sort of say, like, go find this thing and bring it to me. That's really powerful.
- [06:58]
A
Bring it to me and go do X with it even.
- [07:00]
B
Yeah, right, yeah, absolutely. Or I've got meetings coming up tomorrow. I need a briefing book for these meetings. I want to remember what's the background on those meetings in advance of that or what should I be asking in these meetings what's already been like? There's so many things where if your tool has the power to not just, not just index a bunch of files that you have in your file directory, but really be smart about connecting. I can look at your calendar, I can see who you're meeting with, maybe I can see the topic, I can go through your email, I can go through your files, I can put a bunch of things together and just prepare something for you. Even using AI today, that's a lot of steps that somebody's got to take and it'll save you a lot of time if you use AI. But agentic AI is, you know, 3x10x that. So it's really cool stuff.
- [07:53]
A
Okay, next we have John Thompson, overall AI leader, author of a bunch of great books. His most recent one, the Path to AGI is extremely good. Highly recommend that. At the time of our recording, John was the global AI leader for Ernst and Young and we talked about where auditors should really kind of spend their time and prioritize AI and agents. Currently John's the senior SVP and principal of Gen AI program and AI products at the Hackett Group. John's also fantastic follow on LinkedIn. He will. So if you heard the intro, that 63 page PDF I mentioned, he didn't recommend that one necessarily. I don't think that doesn't mean that he disagrees with it. Maybe just didn't come across his feed or in his world. But he will recommend things like that. So if you're uber technical or you want to be, you want to go in at a little more detail level of understanding. John's going to recommend some things like that. Usually it's a 60, 70 page PDF of some research thing. Very, very good. But there's also some kind of quick hitters that he throws out there that are really good and I would say more digestible, easier to read. So follow John as well. He's a very, very good follower when it comes to this and very much one of the premier AI thought leaders in the world.
- [09:14]
C
Agents. Yeah, agents. You're going to hear Aiden agent agents. Nothing but agents in 2025, no matter where you go or who you are. Just like it was all Gen AI for the last two years. It's going to be agents for the next next two years. Agents are going to be a very impactful technology. You do hear from people that, hey, I can't differentiate between a, an LLM and an agent. And that's an issue. That's an issue for all of us. We need to help people understand the difference. There, there will be large language models behind agents, but there'll be lots of other things behind agents as well. So really quick, I'll give you a fast taxonomy on what agents are. Simple agents are pretty much just like an LLM. You give them a prompt, they give you a response. It's pretty much prompt response, prompt response. The only difference really is that an agent can watch the environment. Maybe there's a stock, a filing coming up, or an audit requirement or a policy, you know, a filing requirement or compliance need or something like that. An agent can watch that and say, hey, these things are happening. But we're not getting this step done or it's not going to get done in time. And an agent can jump in the middle of it and proactively do things for you. That's a simple agent. Intelligent agents can do pretty much anything, anything a person can do. They can buy a car, they can spend money, they can, you know, they can bind you to a contract, you know, they can. Intelligent agents can do anything a person can do. And we're, we're building those now. So we have simple agents, we have intelligent agents. We're recording this on December of 2024. So we have intelligent agents now. In the next 18 to 24 months we're going to have something called polymorphic agents, which is really weird word but those are agents that build themselves. So you're going to be able to tell an agent, you know, I want you to do this and that agent will take the prompt and look across the marketplace of pre built agents or building blocks and take those and assemble them into a new agent. Or if those don't exist in a marketplace, the polymorphic agent will build an agent for you. Okay, so it will start from scratch, or you could give it one or two steps, or you can do whatever you want and, and it will go off and build it all. So in the next two years that's what you're going to see transpire in the area of agents.
- [11:29]
A
Next up is Andrew Clark. He's the co, founder and CTO of Monitar. They're an AI governance vendor. So software, AI governance, very, very interesting. Especially if you are in the insurance industry, that's where they really, really specialize amongst some others, but that's where they really specialize. So if AI governance is on your audit program, or even if it's not, you should go at least check out their website, see what they're doing and see if they might be able to help you all. And Andrew has this kind of contrarian view of AI agents and so that was super fun. He has his own podcast, or rather monitor does, and so I was listening to it and he kind of went on this rant about the difficulty in governance with AI agents and how it's near impossible and how we can just use automation and these kind of concepts about around for a long time. And so there's a lot of like, hey, this is more marketing hype than anything. And so the full episode was one of my favorites because we just were able to go kind of back and forth respectfully. And I go, oh, I see it like this. And Andrew go, I see it like this. And then we kind of went back and forth. So that one was actually really fun. If you haven't listened to that, I would highly recommend it. For sure.
- [12:36]
D
Yeah, I want to get into agents a little bit more generally because this is not a new concept. And then I'll describe what, what the technical definition of what agentic AI AI is right now, which also is worth noting when what people are now talking about, as you said on LinkedIn, which I do think a lot of bots have made a lot of these LinkedIn posts I've been seeing lately. But about what, what the common understanding pop culture of what agentic AI is is not actually the definition that IBM, Salesforce, Nvidia, a lot of these real large tech companies, they have a definition of what agentic AI is. So I'm going to give that a definition and then we can, we can kind of analyze and contrast the rest of this episode of like what people are saying is agentic. Oftentimes I don't think is actually agentic AI. And this is based off. This is not my definition. This is IBM definition. Harvard Business Review the general definition we have is really, it's four steps, which is perceive, gather data from multiple inputs from your environment. So you're having an agent is gathering information and then there's a reasoning component. Oftentimes this is where I call reasoning in air quotes. Is this LLM? Like we've just talked about what LLMs do predict the next word, but they've kind of been inserted here as their reasoning now. So, so you're gathering inputs, you're reasoning, you are now acting on those inputs. And this is the thing that gets a little bit scary from a risk perspective when we got to it is they'll actually start making decisions, they'll do something, they'll go book a flight for you or something, make a decision. And then there's a feedback loop as a fourth step of like did I do good or not? And kind of go from there. So that's the definition from Harvard Business, Business View, IBM, Nvidia, Salesforce. A lot of these folks are putting out these articles. That's the definition. However, what we're seeing a lot of times with what people are calling agentic is pretty much sounding a lot of times like automations or now anything that's an automation or even a classical computer program or anything using any sort of a model is now being relabeled agentic. And regardless if some of these are actually really great ideas that I think would be great productivity, answers and good things in business. But I'm calling foul to. Let's be a little bit closer on what is agentic and what's not agentic. And none of that's good or bad. But let's call it what it is because right now the term agentic means absolutely nothing. It just means using a computer program is essentially, essentially where we've gotten to, I think. And there's. Let us let me know how you want to unpack that a little bit more. But really it's those four steps is what I would consider what agentic AI is based on the, the classical definition right now.
- [14:50]
B
Okay.
- [14:50]
A
And is your, I'm going to call it a problem, for lack of better words. But is your problem with the amount of attention on agents, is it the marketing side of it or the realistic use of it or com. Obviously a combination of both. But what is it that's kind of setting you off more so than the others that like no, this is not it. Marketers, will you please stop calling it this or yeah, that is what it is, but it's not. So again, it's not going to be the thing that just has us, everyone on the streets, you know, begging and hunting each other down with sharpened sticks.
- [15:21]
D
That's a very loaded question. It's a pretty much all of the above. But I think, I mean marketing it always this way. Like AI, computer science in general has the best marketing engine around. And then there's a lot of people that maybe somebody puts out something correct and then they'll spit it and Just kind of misinterpret it and then just becomes kind of like lore from there. But I think the two things that are bugging me really are, first off, agentic AI is not new. And everybody has agentic AI, everybody has an iPhone or something equivalent right now. So what about Google Maps? What would you call that? So let's think about that. So you're putting in coordinates. I'm at my house, I want to go to the gym. And I put in the gym because I'm a, I'm a millennial that doesn't know how to drive anywhere without directions anymore. Right. So you put that in and then now it's going to the servers looking at multiple inputs, GPS coordinates as well as other people's telemetric data from their cars and things that are going to Google. Yes, Google is tracking your you when you're using that application. So then taking that information and finding what is the most efficient route for you to go and then updating it dynamically as that changes. Right. And then that's how you go. So we took in inputs, we reasoned, we are executing the instructions and then we're making a feedback loop so that by itself, Google Maps, Apple Maps, whatever your favorite map tool is, is by definition a Gentek AI. So that's not new. And that's using something called multi integer programming, something that dates back all the way back to like World War II and supply chain logistics and things like that. Of like it. We all know when Covid happened, like how the supply chain got snarled up and things like that. Like there's so much that we just take for granted, like how your packages can get overnighted by FedEx or UPS or whoever. Like all of these things are based on this operations research of like route optimization, all these, these, these very fast, efficient algorithms, but they're technically doing that. What we've considered traditionally just stats or operations research or just a computer program have now been rebranded. But where I'm calling foul is we've already established. So there's that we're now saying this concept is new when it's not, and that somehow these horizontal quote unquote agents are now going to, they're going to completely destroy SaaS as a business model. But who's hosting it? That's still SaaS. So anyway, lots of little factual things that I'm calling foul on, but in general we talked about like Google Maps is how that works as an example. Well, now we're throwing in a modeling system that is to predict the next word and we're making that the reasoning component. So instead of using a very optimal decision making algorithm that has been proven and proven time and time again for like route optimization, now we're going to pull that out, we're going to put in a large language model in there versus like why are we putting that in that we have, we have SQL queries we've been using for a long time where you can parameterize those and put in like a coordinate so that like even the fact that we want to, oh, I want to interact with natural language, well that's not new. And we have ways to solve that too. So this is where, and what really gets me is there's some, some recent articles like by IBM and things that are basically saying in the agentic article, you know, for enterprises that haven't found a way to leverage large language models, agentic AI is the reason. So I'm like, whoa, whoa. But what you guys said a couple years ago about what was good about it and I'm not trying to pick on IBM, I'm just like, you guys meaning the industry, right? Yeah, yeah, just the AI industry. That's exactly what you're now saying Agentix going to solve. So it just kind of seems like, oh, large enterprise are realizing that LLMs haven't done everything they wanted. They're just kind of productivity tools. They haven't really been able to go that next step like they were envisioned. So we're going to make up a new paradigm, keep the, keep the CIOs busy for another two years and then we'll come up with something else and just keep the gravy train going. That's what it just versus like all of that put together. If we already have a definition, we have already agents being in place, we have a bunch of misinformation coming around and then like what's different? And have we checked the people that are saying agentic AI is going to be the next new thing? Go back two years in their LinkedIn, if they haven't already deleted it, what did they say LLMs were going to do? And like compare and contrast. That's where I'm kind of, I think as the industry we need to get a little bit better grounding around.
- [19:06]
A
And lastly, I am next and so you'll get my thoughts on agentic AI. All right, thanks everybody for listening and or watching. I know for a fact that multiple internal audit teams have developed and rolled out agents within their internal audit department. And so again, if you're not uber technical then I would find One of these frameworks that you can play around with safely and break them. Like, that's probably the number one thing I hear from people when we really understand, like, why are you not using analytics? There's two reasons. One, they're like, I just don't like it. Like, doesn't make my brain happy. I hate doing it. I get that I'm similar with cyber security. Like, I get the need for it. Super important. Some people love it and that's all they want to do. And I just kind of go, I mean, I need to understand what's out there, but I. I'm not going to deep dive into that. The other one, though, is just fear of breaking it. There was a client we were working with and there was this Excel file with this pivot table in it. And, you know, I was like, oh, well, that's cool. You know, drag this thing over here, click on this. And I said, why have you not, you know, just kind of done this in the past on your own? And she said, well, I didn't want to break it and blow it up. I went, well, if it breaks, you have another copy of the file. Just open it back up. Similarly, go in, break stuff in a safe environment and definitely be okay with that. So, yes, I would absolutely recommend testing and implementing these to some degree. Definitely testing them and playing around with them again in a safe environment. Because it is real. There's definitely hype around this. It is. I mean, every day, regardless of the vendor and the emails that I'm getting, there's something agentic AI somewhere. Multiple people, very much in the AI world have said 2025 is the year of. Of agents. You could search for that and you'll see just article after article after article. CEO or AI expert or leader says 2025 is the year of agents. So it is real. It is here. We definitely need to stay on top of it.
- [21:21]
E
Hey, everyone, thank you very much for listening to this episode of the Audit podcast.
- [21:24]
A
Whatever platform you're listening on right now.
- [21:26]
E
I'm sure there's a subscribe button somewhere, so please hit the subscribe button there. If you're listening through itunes or Spotify, feel free to go give us that five star rating. It only took me about 16 seconds to give myself a five star review, and it really helps to get future guests to come on the show, so we'd really appreciate that.
- [21:42]
A
Lastly, be sure to check out the.
- [21:43]
E
Show notes and follow us on all our social media channels, on Instagram, on LinkedIn and on TikTok. Also, if you're interested, please sign up.
- [21:51]
A
For our weekly newsletter from the Audit Podcast. Thank you all. Have a great one.