Episode Overview

Podcast: The Audit Podcast
Episode Title: IA on AI – Auditing AI in police reporting
Host: Trent Russell
Release Date: October 22, 2025

In this episode, host Trent Russell explores the implications of artificial intelligence (AI) in police reporting, particularly focusing on new California legislation regulating AI-assisted police reports. The discussion weaves in the risks, benefits, and audit implications of integrating AI into crucial public sector processes. Trent touches on best practices in auditing AI-enabled controls, emphasizing audit trails and transparent recordkeeping.

Key Discussion Points & Insights

AI in Police Reporting: Promise and Peril

[00:44]

• Story Highlight: Police departments are now using AI-driven tools to draft formal police reports from body camera audio.
  • Promise: Quicker, potentially more detailed, and accurate documentation.
  • Peril: AI-generated “hallucinations” can lead to significant misreporting and false arrests if left unchecked.

Quote (Trent):

“There's maybe some kind of misdemeanor and there's a hallucination. And now you might get some kind of false arrest for something else or hallucinations somewhere else in this, especially if nobody's reviewing it. There's a ton of issues that could arise from this.” (02:02)

Regulatory Response: California’s SB524

[03:15]

• Law Passed: California enacts SB524.
  • Requirement 1: AI-assisted police reports must be flagged as such.
  • Requirement 2: Agencies must keep an audit trail noting:
    • Who used AI for report generation.
    • What audio/video sources were used.

Quote (Trent):

“The law also requires law enforcement agencies to maintain an audit trail that identifies the person who used AI to create a report and any video and audio footage used in creating the report.” (04:00)

Auditing AI: Best Practices for Internal Audit Teams

[04:20]

• Broader Audit Takeaway:
  • When AI is used in control execution, robust audit logs are essential.
  • Internal auditors should ensure management can consistently produce these logs.

Guidance for Auditors:

• In an AI governance audit or advisory project:
  • Ask: “If you use AI in the execution of a control, can we see the audit log?”
  • Include clear expectations/training on always logging AI prompts and outputs.

Quote (Trent):

“So as part of your—hopefully you've done some kind of AI governance audit or advisory project or you have one coming up soon—it might be worth mentioning the idea of, ‘Hey, if you use AI in the execution of a control, we have to see the audit log.’” (04:50)

How to Set Up and Benefit from AI Audit Trails

[06:10]

• Host’s Practical Example:
  • Always set AI tools (e.g., ChatGPT) to generate audit trails.
    • Shows the logic (“what you were thinking”) and sources.
    • Produces code or analysis traceability as part of deliverables.
  • This approach clarifies outputs and helps verify AI reasoning during audits.

Quote (Trent):

“Whenever you do something for me, give me the audit trail. Tell me what you are—I'm going to put in quotes—‘thinking,’ because they cannot think. Although they say they can think, they can't. But in quotes, tell me what you were thinking and how you came to that conclusion.” (06:40)

• In some custom AI analytics tools:
  • The user prompt, logic, and code are all recorded for full transparency and ease of review.

Quote (Trent):

“I know in some of the agentic AI analytics solutions that we've built, we've built that into it. So when you ask it a question, it gives you that audit trail and says, ‘this is what I was thinking… and basically this is how I got to the result.’” (07:58)

Tangential Resource Recommendation

[09:55]

• Instagram Shoutout:
  • “Omani Auditor” (@omani_auditor) is a recommended follow for interesting, consistent audit content.

Notable Quotes and Memorable Moments

• On the risk of AI-generated hallucinations in law enforcement:

  “This isn't like writing the first draft of your English paper. There's like real world, potentially long-term consequences for the individual if there's a mistake made in getting the police reports written by an AI tool.” (02:45)

• On expectations for transparency in internal controls:

  “If you use AI in the execution of a control, we have to see the audit log. That could be something as far as training goes, it could be pretty simple. It could be a—when you use it, including your prompt—‘Hey, AI tool, in case I get audited for using this in this manner… please include an audit trail.’” (05:20)

Timeline of Key Segments

| Timestamp | Segment | |-----------|--------------------------------------------------------------------| | 00:44 | Introduction to AI in police reporting and its emerging usage | | 02:02 | Risks of AI hallucinations and long-term consequences | | 03:15 | California's SB524 legislation overview | | 04:20 | Audit trail requirements and their implications for internal audit | | 06:10 | Host’s best practices for AI audit trails and transparency | | 07:58 | Custom AI solutions and maintaining an audit trail | | 09:55 | Instagram recommendation: Omani Auditor |

Summary Table: Audit Considerations for AI in Police Reporting

| Control Area | Risk | Best Practice | |--------------------|------------------------------|---------------------------------------| | AI-generated Docs | Hallucination/False Data | Mandatory human review; audit labeling| | Audit Trails | Accountability gaps | Identify user & data sources tracked | | Transparency | Opaque AI reasoning | Require AI to log ‘thinking’ steps | | Training | Inconsistent audit logs | Simple mandatory process/checklist |

Final Takeaways

• AI in high-stakes environments like police reporting raises significant risk if not closely monitored.
• Legal requirements, as in California’s SB524, set a precedent for transparency standards.
• Internal auditors should champion robust, user-friendly audit trail expectations for all AI-enabled controls.
• Simple steps—like requiring AI tools to record their ‘thought process’ and source code—can dramatically improve auditability.

This concise, topic-rich summary covers the core themes and practical audit guidance discussed in the episode, making it accessible for practitioners or listeners who missed the episode.