← The Audit Podcast
The Audit Podcast
Ep 264: How Internal Audit Teams Are Building AI Tool with no Tech Background Needed (Snowflake)
00:34:31
Loading summary
Transcript40 lines
- [00:00]
A
We're in the age of AI and so I encourage all the auditors tuned in for, you know, I encourage you all to focus on being change agents and look for various ways to leverage data and AI tools in doing more with less. Because, you know, the biggest challenge that you often face in internal audit is how can I do more with less?
- [00:27]
B
Hello, everybody. Welcome to another episode of the Audit Podcast. I'm your host, Trent Russell. Today on the show we have a few folks all from Snowflake. And so we're going to start with Amrita. Amrita is the head of internal audit at Snowflake. For those that don't know, because some context here is going to help throughout the episode. Snowflake is a just think data warehouse. So lots of organizations use Snowflake and pull data from it, run analytics against it, all that kind of good stuff. And so while we're on the topic, what we're going to be talking about with Amrita and her team are a few use cases where they've used functionality or tools native to Snowflake. So if you're a Snowflake customer, they're going to walk you through some AI and data use cases that their internal audit team has built using the same tools that you have access to if you're a Snowflake customer. We also have Pooja who joined us to walk through the use case that she built. And if you're looking at my screen, this is going to be one of those where you're probably going to want to watch this on YouTube because there's a lot of screen sharing and talking through and showing exactly what it is that they built. Something I want to point out though, we do ask it, but I'm going to and we get to the point much quicker during the episode than what I'm about to do, but to kind of prove this out. So if you look at. I'm going to scroll down and we're going to take a quick peek at Pooja's experience experience here on based on her LinkedIn profile. So she was a tax analyst, audit senior assistant, treasury intern, financial advisor, equity research analyst, graduate teaching assistant, financial management, another grad assistant position and assistant securities analyst. Now it's stuff like an internal audit. So from all that you did not hear and if you go look at our profile in more detail, you will not see where there is any, hardly any much mention of an I T background, certainly no data analytics background, AI background, high technical background, anything like that. Pooja is an auditor without the tech background and the Reason to point that out is because what she built, the AI tool that she built, which is awesome, I believe she said, took her an evening to build. I don't want to give away too much of what actually it is. So it's really hard for me not to do that right now because it is awesome. But she has no technical background and she built this thing in an evening. Reason I point that out, some of these things can be built this easily without a technical background. Word of caution. There's a lot of other AI use cases that you cannot execute unless you do have a high, high level of technical competency. So that's something that I'm starting to pick up on a little bit as we're talking to folks, audit teams is they go, oh, can't we just take this and this and then put it into ChatGPT or Copilot and it'll spit out the answer this way and it'll be perfect and beautiful and you know, there's a lot more to it than that for that use case. So not only having the list of use cases is going to be important for everyone, but then also tying the competencies and the resources that you need, the tools that you need in order to make that use case work for you and your organization is very, very critical in terms of a feasibility assessment. And the bigger the team, the more complex that's going to be. So there are some use cases where these really, really large teams have this full agentic infrastructure that they have to use, but then for a smaller team, they're able to use maybe just Copilot or just the agents in there to do that. So a lot of this depends on the team. However, again, if you have Snowflake, you can definitely do this. Lastly, we have AJ who is an internal audit manager at Snowflake and AJ is going to walk through some of the AI use cases and the data analytics use cases also. With all that said, here we go.
- [04:42]
A
Okay, great. So really nice to see you, Trent. I'm joined here by my team, Ajay and Pooja. They both oversee operational audits and SOX testing in key areas on the IT and business side. So in my current role, which is pretty different from a lot of the traditional CAE roles, I deal a lot with sales as well and talk to our customers on how we're leveraging Snap Snowflake or Audit Transformation and what we're doing in driving proactive insights for our stakeholders. Now, for those of you who do not know what Snowflake does, it is a AI cloud based data warehouse that provides a single platform for storing and sharing data, you know, along with enabling querying capabilities for data analytics and AI solutions. So as Customer Zero, we are always very excited to to leverage our product for driving efficiencies in the way we carry out our testing and provide these insights to our stakeholders. So yeah, we thought we could start this session today by showcasing some demos to get the audience excited about how we continue to transform the internal audit profession through use of AI and analytics. And maybe Ajay and Pooja can showcase some of these use cases for you. We have a handful of those. Some where we are leveraging AI capabilities for driving efficiencies and then some where we're using analytics for continuous monitoring to expedite our testing efforts. So with that I will hand it over to Pooja to first showcase what we've been able to do with using Doc AI. It's document intelligence capability within Cortex, which is our AI platform within Snowflake. And we've been able to use this capability for testing control around looking at non standard terms and revenue contracts.
- [06:42]
B
So you mentioned docket, Cortex and Snowflake. Can you get Snowflake? I think we're all pretty good with Cortex and Docker. How do those sit? What do those do?
- [06:52]
A
Cortex in a nutshell is basically our AI platform which houses all of these AI solutions and capabilities that we provide for enterprise data that resides within Snowflake. So it has things like LLM functions for summarization, text classification. It also has Cortex Analyst that does text to SQL queries for conversational data analysis. It has a lot of other things like Cortex search for knowledge retrieval in documents and then Doc AI. What I mentioned is kind of like your document intelligence feature. It's like OCR, right? Using LLMs for automated data extraction and docs with a UI that's very business user friendly. And so that's what you know, Pooja being on the business side was able to play with to see how best to use for controls testing.
- [07:45]
B
Okay, so if you're a Snowflake customer, you could have access to Cortex and Doc AI. This isn't limited to test your team. Okay, perfect.
- [07:51]
A
It's out of the box. Yep.
- [07:53]
B
Okay. All right, Pooja, we'll throw it to you.
- [07:55]
C
Okay, great. Thank you Amrita and great to be here. Trent. I think every audit professional knows the feeling of like drowning in documents. I think contract review control is exactly like one of such controls where you have to scan like multitude of documents of varying shapes and sizes. It is a pretty foundational Control for all companies. But it's massively like a mambo bottleneck for companies too. What if I tell you that you could train like an AI first year analyst to review thousands of these contracts with perfect consistency and superhuman speed? And I'm going to show you exactly how we do that at Snowflake. We've leveraged Snowflake AI to fundamentally transform this control. We've replaced manual effort with machine certainty and just to quickly move over to the ui. So this is what the UI for Doc AI looks like. You can basically go ahead and upload like a document which could be a PDF, all of the supported formats, which are many in this instance. And once you've uploaded, you can basically go into this particular thing and the tool looks like this, where you can basically go in and ask questions. In this instance you can basically ask questions like, hey, is there a price lock, is there a discount lock? Anything that impacts revenue or is revenue impacting? And once you upload a document, you can basically go in and ask questions in natural query language, meaning that you can ask questions around volume discount, like is there volume discount, is there a price cap, is there a price cap percent anything that is relevant when you are assessing contracts for revenue assessment purposes. So the key to this tool here is that it's an OCR tool at heart, meaning that it never hallucinates or makes up data outside of the order form. Secondly, it is a zero shot model, meaning that this model is smart enough to kind of pick data immediately without any upfront training to increase or enhance the audit trust. Notice the confidence indicator score here. This also helps us understand and evaluate how confident the model is when it picks up answers. All of this is possible in a zero coding business friendly UI right now. And then to add onto it, there is also model accuracy indicator which builds up as and when you keep training it for us, currently it's at 99% which is pretty good.
- [10:51]
B
Not bad.
- [10:51]
C
Yeah, yeah. And then the power of like Snowflake platform does not even end here. It's because all of this data or the valuable insights that you garnered from these contracts, you can basically get this unstructured data into a structured format, meaning that you can pull all of this data in a tabular format. Again, a business user friendly plus initiative. As you can see, this is a table that you can basically have all of those unstructured data kind of like pulled in. It's pretty user friendly. If you give it to a business user, they can basically use this table to further analyze or draw conclusions out of this data.
- [11:38]
B
And so the one that we were just looking at, that's your example of one contract. But then if you put 50 contracts in there, you would have 50 records in there. And each kind of field or header or column name is basically a question that you've asked of the contract. And then it just like pulls that out and spits it right there.
- [11:58]
A
Yes.
- [11:59]
C
Just in an easier format.
- [12:01]
B
Yeah, nice.
- [12:03]
A
And I guess just from an impact perspective, we were able to analyze 7,000 plus contracts within 15 minutes. And the impact here is that we have order management team that's about seven to eight people that you know and part of their job is to review these contracts for extraction of non standard terms. Obviously 7,000 I said is below a certain threshold, but from a socks perspective, Pooja was able to use document AI to actually even highlight certain contracts that humans missed. And now, you know, internal audit is being a catalyst for sparking interest in firstline to also leverage AI. Perfect. Which is the beauty here in terms of like how, you know, technology can be embraced for these kind of use cases.
- [12:53]
B
Okay, I know the question. Every time we see a cool demo like this, the questions that we always get, there's a few that we always get. One Pooj is your background in like it and you have some kind of, you're a data science super genius and that's how you built this.
- [13:06]
C
Nothing. So it's a zero coding like I said. Again, it's really important to reinstate the fact that this is absolute zero coding business friendly user interface where you can go in and ask questions. So similar to like how you would train a first year analyst sitting right next to you and it kind of like just does the job. And just like Amrita mentioned, it's turning weeks of manual work into like a half day affair because we process like these 7,000 contracts in almost like 30 minutes and stuff like that.
- [13:37]
B
Okay. The other question that we always get, how long did it take you to do this? Resources, all that kind of good stuff.
- [13:44]
C
I think it was just me and evening's worth of time to train this cool thing that we have. And I think within, I think a half day worth of work to kind of train it, get it to that accuracy. Because like I said, it's a zero shot model. It's pretty smart, extracts data quickly and you can basically get started or pass the contract almost starting the next day.
- [14:12]
A
Cool. So I think now we should show if Ajay, you can be quick in this. Also want to show some other demos on how we're leveraging AI, but also analytics for ITGCs and socks. So one use case that'll resonate with everybody is SoC1 reports. We spent a lot of time in reading these exhaustive reports and then updating templates and we thought that would be a great use case for leveraging AI to do that. So that's what Ajay is going to showcase now.
- [14:42]
D
So yeah, as Amrita was just mentioning, I think I'll walk through a couple of use cases on how on the internal audit team we are pretty much trying to transform our SOCKS testing and where the goal is to maximize our assurance and minimize the testing effort that will be required going forward. So the first use case I want to talk about is on the SOC Report Automation. The problem statement is pretty much that every year manually we get SOC reports for all our vendors for SOC reports and we have to look at like any control failures and whether that impacts our like, you know, Snowflake as a customer. So we have to draft like manual templates and fill out, which is pretty much a copy paste effort that requires like 8 to 10 hours per report. And if you think like there's 20 vendors, that's around 200 man hours. So what we did was we leveraged our Snowflake LLM models to basically be able to kind of put in PDF and it automatically scans through the whole PDF and gives us the important information we need in a matter of minutes to be able to just put it into a template in a downloadable format. So as an example, this is something that I preloaded here, but you put in the SOC report in the drag and drop and then it gives you all the information that typically someone would manually go in and kind of look and copy paste. But here is just all available at one glance so you can see like what the SOC report period coverage is, what sort of subservice organizations are there, what are the complementary user controls that as a company we should be having at our end and seeing if we have coverage over it and what are all the different control objectives and their outcomes and if there are any exceptions in the report that were noted, what was the management response for those SOC reports? So all of it is like available at a, like in a single glance for you to quickly look through and like get your evaluations done, which we anticipate will save around 150 to 160 Manas for a 20 vendor SoC report.
- [16:52]
B
Is it possible? Have you guys done this where. Because I know this is always such a pain. But then when you have those subservice ones and you have to go get the subservice SOC report and then that can just this crazy rabbit trail that you have to go down. I'm assuming it helps a ton with that.
- [17:07]
D
Yes. And that's I think like one additional enhancement like this is like the basic use case so it reduces your effort. But and more advanced that we are still kind of enhancing it is to make sure that if there is subservice organizations and it should automatically look into the existing list of reports we have and tell us if there are any other subservice orgs that we might have to request a SOC report. That will help in cases where you have a new SOC report and you are evaluating it for the first time and you might have a subservice which you are not already covering at your end. But yeah, that's totally in the works and something we want to make sure it's incorporated as well. Then moving on to the other use case, this is more towards how we leverage data analytics from our snowflakes capabilities to automate our key reports, which is a significant chunk that we test manually every year to make sure the reports used in different business processes are complete and accurate. So instead of manually doing that, I think we took the biggest use case for us, which is workday, where we have a significant number of reports. And what we did was we kind of ingest the audit logs for each of those key reports and put it into our snow house, which is our database, and basically perform analytics to one show see that how many such reports are there where there are no changes that have been made since the last time it was tested. Because for those you could typically benchmark and say that because there are no changes, you don't need to perform like a completeness and accuracy testing necessarily if you have done that in the year before. So that will help going forward to benchmark and basically reduce the overall testing effort. Now additionally in this, what we have also done is if there are any calculated fields in your report, typically external auditors also want to know that is there a change in those attributes and what is the logic on that? You can basically also see at attribute level if there are any calculated fields that you might have if any true changes have been made. If not, you could potentially benchmark those as well and reduce your testing effort. Because for these calculated fields you have to manually go and get those screenshots or manually get at a field level the audit trail. And this automated fashion makes it way more easier and reduces significant testing efforts. If you think about like 50, 60 key reports.
- [19:39]
A
And also the number of screenshots that you guys have had to share with external auditors goes away. No one likes those screenshots. No worse.
- [19:48]
B
No worse.
- [19:49]
A
Yeah, yeah.
- [19:51]
D
So, I mean, we are pretty excited to kind of make this into use in our upcoming testing cycle and see how much hours it saves and which can be then, you know, used for other audits and operational audits as well. The third use case I want to quickly talk about is the termination control testing. So this is the third use case, which is around access termination, which is typically a very big control across industry in that sense. So we have single sign on at our place, which is through Okta. So we have one big termination control that we test, which as usual, it's manually being done before. But the challenge there was also Okta decided to be like, they will only have the logs being retained for 90 days in the system. So that kind of creates a compliance challenge because if you have to test it, you either have to time yourself to go and download the report at the right time or ingest those into a database so that you can retain them and have, you know, put control so that you have comfort over the completeness and accuracy of that data. So we ingested the Okta logs into our snowhouse database. And, you know, with that we have two sets of dashboard here. The first one is more specifically tailored to management because for management, operationally, the threshold is that delayed termination should not be more than a day. So the moment the last working day gets kicked in, it should automatically trigger the termination. And if it doesn't happen, it should automatically trigger an alert. And for SOX purposes, we have that defined to five days. So that is like additional sort of like visual that we have specifically for our SOX testing, which we have added onto this, where we can see, like, based on our threshold, are we seeing any cases where there's a delayed termination which could then potentially be investigated for whether it's a false positive or not? But this significantly cuts down your time from manually looking at each and every user and the last working day and confirming whether that's a valid use case or not. So with this, we hope to kind of reduce our testing time as well. And once this works well for us, the eventual plan is to kind of turn it over into a management monitoring control so that it can be a continuous monitoring control rather than a point in time control where we go in for a specific period and test that.
- [22:22]
A
Yeah, I would say not just from a testing perspective, but even from a risk perspective. Right. This is continuous monitoring of this control versus relying on auditors to also come and do point in time testing. So at any given point in time management can also see whether you know, this is a healthy percentage or not. And they can see operating effectiveness, you know, over the course of time here clearly. And to the extent something drops below their sla, they can investigate. Click on this to see the details. You've kind of parsed this out into two categories I would say one is an IT breach versus HR breach because more often than not you, you'll realize in these controls, right, the people managers are also not having good hygiene on keeping work DHCM up to date. And so some of those things can, some of this monitoring over time can even help you in offering the right level of trainings or processes to be put in place for more proactive mitigation. So yeah, it really helps on both fronts.
- [23:22]
B
I think the burning question that the listeners are thinking right now is Amrita, are you hiring so that people who want to play with this kind of cool stuff and build this kind of stuff will be able to do so?
- [23:35]
A
Yes, absolutely. I mean I actually think skills are really converging. So I'm even pushing my business auditors as Pooja to closely understand these capabilities and drive efficiencies there too. I think each one of us can invest the time in learning some of these capabilities and seeing how best we can utilize it, you know, in our day to day activities.
- [24:02]
B
It sounds like from, from what Pooja said that there's really not like the, the learning curve isn't super steep where it's having to spend months and months and months to build some of this stuff out. I mean what was it, 7,000 contracts and you basically built it out in less than a day is what it sounded like.
- [24:18]
A
Yeah, very nice.
- [24:20]
B
I know a lot of the questions that we've been getting around this of like hey, we have this AI use case and it's going to test a control this way or it's going to do CM or CA this way. When you guys are coming up with use cases, how kind of lock and step, especially on the sock side do you feel like you have to be with external audit to go if we're going to build this, are you guys cool with it? This is like this could be the output. This is what the expectation is that they've been pretty okay with working with you all to make sure. And the reason I asked this, I know people would ask me, hey, we want to test 100% of this population and so we're going to build it and then our external auditor is going to love it. And I go, what? Hold up. You go ask them first if they're cool with the way you guys are going to start doing testing, if you're going to do it this way. So I was curious what the relationship is between, between you all and external audit to make sure that you don't build something that you go, that's cool. We can't really rely on that.
- [25:11]
A
Yeah. So, you know, for us, in terms of audit transformation trend, how we're looking at it is really in three pillars in our team. In terms of like, what are good use cases for analytics? What are good use cases for AI agents to help solve? And then document AI is another piece like document intelligence. Right. For unstructured documents, how can we expedite that instead of relying on humans, which is where Doc AI and what Pooja showed comes into play. So depending on these three pillars, we kind of fit the use cases into those, I would say for analytics broadly. I'm a big fan of analytics. Why? Because you can test 100% of the population, right. You can get proactive insights. It could be continuous auditing versus waiting for a point in time to figure out, oh, what's the impact now? Like a lot of work, which is after the fact that no one really likes. So I think it gcs there are great use cases for continuous monitoring and analytics. What happens with analytics often though is it takes time to build, right. You need to ingest the data, you need to make sure good pipelines there. You do need to work to build the logic for the key attributes or things that you want to look at or risk insights. Right. So that does take some time, which is why we do that on a case by case basis. AI agents are great because you can replace humans with like expediting things. They can also do continuous monitoring. I get it. But like, again, it all comes down to data. So for now, AI agents, we're really leveraging for transactional controls. You know, things where people are testing like 25, 30 samples, right. Getting a lot of documents and then spending a lot of time in updating templates. So for a lot of those transactional controls in our program, we're exploring, we're actually exploring a design partnership with a startup company and they're actually working to build a native app within Snowflake for us. Because a lot of our data lives in Snowflake. So the plan is once that's built, we can install it within our own Snowflake account and use it for testing socks controls. But it's going to be dependent on the fact that we upload the PVCs, we give the agent the template and then it's pretty much like instead of giving it to someone in an offshore team, we can have an AI agent do that for us more quickly. So that saves the team some time and focusing on more value added insights or maybe operational audits versus really that exhaustive documentation aspect. And on ITGCs, to the extent we can do analytics, we're really trying to push for that. If we have the data in Snowflake and we have a good appreciation of, you know, what are the key policies around the data, we want to build analytics and drive more continuous monitoring. So that's kind of how we're broadly thinking about it.
- [27:50]
B
I think there's still the fear of we bring in all this AI and it replaces our auditors. And I'm still not hearing that from audit leaders of like that's the goal or anything to that, to that point. But how are you shifting resources? Maybe I'm thinking short term and then long term. So is it, hey, now that Pooja developed this crazy thing that is going to save us 10,000 hours or whatever the case is, now she can either go build something else or she can focus more on advisory projects. What does that look like?
- [28:21]
A
Yeah, and I guess I think we all have to appreciate the fact that building things takes time. So it's always a journey and I think you always want to build things where you feel like you get the most roi. And at the end of the day, I think more than thinking about I'm going to use AI everywhere, I think you should think about what is the problem that I'm trying to solve here. And sometimes AI is probably not the answer also. So I think it's really important for people to hone in on what is the problem and what is the impact that I want to drive and then figure out what's the capability or the tool that I want to use to solve that. Because sometimes existing systems can solve those problems too. I know AI is this magic thing that everyone keeps talking about right now and it's great. Don't get me wrong. I absolutely think that everyone needs to embrace it because if you don't, I think you will be left behind because it does really drive a lot of efficiencies and takes away a lot of redundant, you know, you know, non value add, you know, tasks from your plate. But it has to be used for the right use cases. You always have to think about the benefit and the impact that I'm trying to drive and then, and then work accordingly. And then to answer your question on resource replacement, I don't think that's the case. I actually think it's going to augment our work. Right. And jobs will evolve. I mean we've all seen this through Internet coming into play, going to the cloud coming into play. I know this is a much bigger wave, but I do think we all need to embrace this. There is an opportunity. And actually the flip side of this is there are risks too. And in internal audit I think it's important for us to also understand what are the risks that associated with AI. So we can also play that trusted advisory role for our stakeholders to make sure they're better educated on what controls to put in place as well. So I think it's great for internal audit profession and you know, we should wholeheartedly embrace it.
- [30:21]
B
I think one of the things we get asked the most also is what are the AI use cases. We don't know where to start with the use cases and I think the three that you showed were incredible. And so for folks that are looking for some kind of inspiration, especially if you are snowflake customers, those are some pretty rock solid ones right there that you can start with. But with that said, Amrita, I'm going to throw the mic to you. What do you want to leave the audience with today?
- [30:46]
A
Yeah, I guess firstly thanks Trends for giving us the opportunity to showcase what we've been able to do. It's all about knowledge sharing and I think all of us should come together to do that. But I guess since we're on this topic of AI transformation, I would say we're in the age of AI and so I encourage all the auditors tuned in for. I encourage you all to focus on being change agents and look for various ways to leverage data and AI tools in doing more with less. Because the biggest challenge that you often face in internal audit is how can I do more with less? Produce more audits, usually with less resources, produce more insights for our stakeholders, more proactively versus reactively. So it's really encouraging you guys to think about how we can move away from traditional sampling ways. I keep pushing my team on that and move to more automated AI and data driven techniques. Not rely on people. Right. But allow for data to tell the story. I definitely believe that the future of audit probably is here today too is data driven. And so it's very important to have that data driven mindset. And then obviously you know, be able to contextualize insights, you know, with good business acumen. Because data doesn't matter if you don't have a good understanding and appreciation of your business strategy and objectives. So, yeah, that's broadly on the, I would say professional side and then on the personal side, you know, I would also say lead with empathy, you know, invest in relationships both professionally and personally. I think it's very easy for all of us to get consumed by technology. I sometimes, you know, I'm guilty of that myself and rely on all sorts of AI assistance. But I think it's also important to take a break to also socialize and interact with your loved ones and, you know, build and foster genuine connections. So, yeah, stay hungry, you know, develop your skills. For me personally, you know, I'm, I'm trying my best to always juggle multiple priorities and see how best can I spend time with my 21, one month old daughter. And in fact, you know, Trent, I know you'd asked me before what is in my browsing history or chatgpt or what do I do in my Q and A with Gemini. That's what sometimes I do. I get parenting advice, you know, how do I deal with tantrums, you know, what do I do when my kid does this? So that's what I'm also focusing on, you know, personally is how best to develop my parenting skills too, especially learning a lot of patience. So, yeah, on how to be a kid again with my kid, which is fun. So, yeah, so definitely, you know, enjoy with technology, but take a break and really, you know, invest in relationships as well.
- [33:53]
B
Hey everyone, thank you very much for listening to this episode of the Audit Podcast. Whatever platform you're listening on right now, I'm sure there's a subscribe button somewhere, so please hit the subscribe button there. If you're listening through itunes or Spotify, feel free to go give us that five star rating. It only took me about 16 seconds to give myself a five star review and it really helps to get future guests to come on the show, so we'd really appreciate that. Lastly, be sure to check out the show notes and follow us on all our social media channels, on Instagram, on LinkedIn, and on TikTok. Also, if interested, please sign up for our weekly newsletter from the Audit Podcast. Thank you all. Have a great one.