IA on AI – Auditing AI in police reporting

Summary

---

Episode Overview

Podcast: The Audit Podcast
Episode Title: IA on AI – Auditing AI in police reporting
Host: Trent Russell
Release Date: October 22, 2025

In this episode, host Trent Russell explores the implications of artificial intelligence (AI) in police reporting, particularly focusing on new California legislation regulating AI-assisted police reports. The discussion weaves in the risks, benefits, and audit implications of integrating AI into crucial public sector processes. Trent touches on best practices in auditing AI-enabled controls, emphasizing audit trails and transparent recordkeeping.

---

Key Discussion Points & Insights

AI in Police Reporting: Promise and Peril

[00:44]

• Story Highlight: Police departments are now using AI-driven tools to draft formal police reports from body camera audio.
  • Promise: Quicker, potentially more detailed, and accurate documentation.
  • Peril: AI-generated “hallucinations” can lead to significant misreporting and false arrests if left unchecked.

Quote (Trent):

“There's maybe some kind of misdemeanor and there's a hallucination. And now you might get some kind of false arrest for something else or hallucinations somewhere else in this, especially if nobody's reviewing it. There's a ton of issues that could arise from this.” (02:02)

Regulatory Response: California’s SB524

[03:15]

• Law Passed: California enacts SB524.
  • Requirement 1: AI-assisted police reports must be flagged as such.
  • Requirement 2: Agencies must keep an audit trail noting:
    • Who used AI for report generation.
    • What audio/video sources were used.

Quote (Trent):

“The law also requires law enforcement agencies to maintain an audit trail that identifies the person who used AI to create a report and any video and audio footage used in creating the report.” (04:00)

Auditing AI: Best Practices for Internal Audit Teams

[04:20]

• Broader Audit Takeaway:
  • When AI is used in control execution, robust audit logs are essential.
  • Internal auditors should ensure management can consistently produce these logs.

Guidance for Auditors:

• In an AI governance audit or advisory project:
  • Ask: “If you use AI in the execution of a control, can we see the audit log?”
  • Include clear expectations/training on always logging AI prompts and outputs.

Quote (Trent):

“So as part of your—hopefully you've done some kind of AI governance audit or advisory project or you have one coming up soon—it might be worth mentioning the idea of, ‘Hey, if you use AI in the execution of a control, we have to see the audit log.’” (04:50)

How to Set Up and Benefit from AI Audit Trails

[06:10]

• Host’s Practical Example:
  • Always set AI tools (e.g., ChatGPT) to generate audit trails.
    • Shows the logic (“what you were thinking”) and sources.
    • Produces code or analysis traceability as part of deliverables.
  • This approach clarifies outputs and helps verify AI reasoning during audits.

Quote (Trent):

“Whenever you do something for me, give me the audit trail. Tell me what you are—I'm going to put in quotes—‘thinking,’ because they cannot think. Although they say they can think, they can't. But in quotes, tell me what you were thinking and how you came to that conclusion.” (06:40)

• In some custom AI analytics tools:
  • The user prompt, logic, and code are all recorded for full transparency and ease of review.

Quote (Trent):

“I know in some of the agentic AI analytics solutions that we've built, we've built that into it. So when you ask it a question, it gives you that audit trail and says, ‘this is what I was thinking… and basically this is how I got to the result.’” (07:58)

Tangential Resource Recommendation

[09:55]

• Instagram Shoutout:
  • “Omani Auditor” (@omani_auditor) is a recommended follow for interesting, consistent audit content.

---

Notable Quotes and Memorable Moments

• On the risk of AI-generated hallucinations in law enforcement:

  “This isn't like writing the first draft of your English paper. There's like real world, potentially long-term consequences for the individual if there's a mistake made in getting the police reports written by an AI tool.” (02:45)

• On expectations for transparency in internal controls:

  “If you use AI in the execution of a control, we have to see the audit log. That could be something as far as training goes, it could be pretty simple. It could be a—when you use it, including your prompt—‘Hey, AI tool, in case I get audited for using this in this manner… please include an audit trail.’” (05:20)

---

Timeline of Key Segments

| Timestamp | Segment | |-----------|--------------------------------------------------------------------| | 00:44 | Introduction to AI in police reporting and its emerging usage | | 02:02 | Risks of AI hallucinations and long-term consequences | | 03:15 | California's SB524 legislation overview | | 04:20 | Audit trail requirements and their implications for internal audit | | 06:10 | Host’s best practices for AI audit trails and transparency | | 07:58 | Custom AI solutions and maintaining an audit trail | | 09:55 | Instagram recommendation: Omani Auditor |

---

Summary Table: Audit Considerations for AI in Police Reporting

| Control Area | Risk | Best Practice | |--------------------|------------------------------|---------------------------------------| | AI-generated Docs | Hallucination/False Data | Mandatory human review; audit labeling| | Audit Trails | Accountability gaps | Identify user & data sources tracked | | Transparency | Opaque AI reasoning | Require AI to log ‘thinking’ steps | | Training | Inconsistent audit logs | Simple mandatory process/checklist |

---

Final Takeaways

• AI in high-stakes environments like police reporting raises significant risk if not closely monitored.
• Legal requirements, as in California’s SB524, set a precedent for transparency standards.
• Internal auditors should champion robust, user-friendly audit trail expectations for all AI-enabled controls.
• Simple steps—like requiring AI tools to record their ‘thought process’ and source code—can dramatically improve auditability.

---

This concise, topic-rich summary covers the core themes and practical audit guidance discussed in the episode, making it accessible for practitioners or listeners who missed the episode.

Transcript

[A]

Welcome to the IA on AI Podcast, part of the Audit Podcast Network where we bring you weekly updates on AI from the internal auditor's perspective. Here we go. Hey everybody. We are back after a week off the October audit conference. Busy season got us a little bit over the past couple weeks, but we are back and should be back for the rest of the year. And this week we only have one story to share from phys.org that's p h y s.org concerns about AI written police reports spur states to regulate the emerging practice. So from the article, police are getting a boost from AI with algorithms now able to draft police reports in minutes. The technology promises to make police reports more accurate and comprehensive as well as save officers time. Probably can already see the risks. I mean, as soon as I saw this I was like, oh, this is going to be awful. Okay, so it goes on. The idea is simple. Take the audio transcript from a body camera worn by a police officer and use the predictive text capabilities of large language models to write a formal police report that could become the basis of a criminal prosecution. So immediately you can see the risk. I'm going to tie this into internal audit. But we also have, there are internal audit departments within large police organization. So I don't know if you know those are listening right now, probably not very many, but if so, I'm sure they're aware of this. But either way, there is again an internal audit tie that I'm going to bring this back to. But you can see the risk in this. Hallucinations happen. There's maybe some kind of misdemeanor and there's a hallucination. And now you might get some kind of false arrest for something else or hallucinations somewhere else in this, especially if nobody's reviewing it. There's a ton of issues that could arise from this. And the article talks about, hey, this isn't like writing the first draft of your English paper. There's like real world potentially long term consequences for the individual if there's a mistake made in getting the police reports written by an AI tool. So this is California specific. So SB524 went into law last week requiring all AI assisted police reports to be marked as being written with the help of of AI. The law also requires, here's the audit part. The law also requires law enforcement agencies to maintain an audit trail that identifies the person who used AI to create a report and any video and audio footage used in creating the report. And where I'm like tying this back to the broader internal audit Risk landscape is when management is using AI in the execution of a control. That's been one of the hot topics over the past probably year is how do we audit these controls where management's using AI? So as part of your hopefully you've done some kind of AI governance audit or advisory project or you have one coming up soon, it might be worth mentioning the idea of hey, if you use AI in the execution of a control, we have to see the audit log. That could be something as far as training goes, it could be pretty simple. It could be a when you use it, including your prompt hey, AI tool. In case I get audited for using this in this manner by our internal audit department, please include an audit trail or just please include an audit trail. So for example, when I use chat GPT by default I have it give me the audit trail. So I went into basically my profile and said, hey, whenever you do something for me, give me the audit trail. Tell me what you are. I'm going to put in quotes thinking, because they cannot think. Although they say they can think, they can't. But in quotes, tell me what you were thinking and how you came to that conclusion. List the sources. If you run some kind of analysis, a data analytic, something to that effect, then show me the code that you use to develop that. If nothing else, if it's extremely interesting to see kind of behind the scenes of how the tools work, when you see it go, oh, I did this because of this, this and this and this. And hey, I'm thinking about this. The user asked for this. I think they meant this. Nope, actually I think they meant this. And then to see the output that ultimately comes with that. I know in some of the agentic AI analytics solutions that we've built, we've built that into it. So when you ask it a question, it gives you that audit trail and says this is what I was thinking. And basically this is how I got to the result that I'm now giving. As well as if it produces code or produces an analysis or an output, it'll produce the code also. So that that is auditable. One more thing unrelated to AI specifically, although the topic does get hit on for any of those on Instagram, highly recommend following that Omani Auditor. If you're looking at my screen you can see that's that underscore Omani O M a n I underscore Omni Auditor. This is an account that Abbas Al Awati has been running for I think four or five years, pretty consistently so again, if you're an Instagram person looking for some audit content on Instagram. Highly recommend that Omani auditor Thank you for listening, and be sure to follow the link to greenskiesanalytics.com in the show notes and schedule time to see how green skies can make the hype of AI a reality in your internal audit department. All right, that's it for this week. We don't really have a catchy slogan to end the show yet, so if you have one and you want to send it to us, we'll be happy to include it. And if we get a bunch, we'll just do a different one every single time. But until then. Well, I don't know until then, because we don't have anything to leave the folks with yet. So have a good week.