Governing Generative AI at Scale: From Model Risk to Systemic Control

Summary

---

Podcast Summary: Governing Generative AI at Scale – From Model Risk to Systemic Control

Podcast: RSAC
Hosts: Tatiana Sanchez & Casey Zerkis
Guest: Varun Raj, Cloud and Engineering Executive
Date: April 7, 2026

---

Episode Theme & Purpose

This episode delves into the evolving challenges and strategies for governing generative AI at scale. The discussion centers on moving beyond traditional, static approaches to governance, advocating for dynamic, architecture-embedded controls that address not only model risk but also systemic risk. Guest Varun Raj shares insights on reframing AI governance as a runtime property, designing robust control planes, and ensuring organizations' AI systems are safe, trustworthy, and auditable in production environments.

---

Key Discussion Points & Insights

1. The New Paradigm of AI Governance

[00:29]

• AI governance fundamentally differs from traditional technology governance.
• Many organizations mistakenly treat AI governance as a compliance or documentation exercise.
• AI systems bring unique, behavioral risks that are dynamic and require real-time oversight.

“AI governance is more dynamic and it needs to move beyond just a checklist mindset.” – Casey Zerkis [00:29]

2. Runtime Governance for Dynamic Systems

[03:00]

• Traditional governance is static: policies, pre-deployment reviews, documentation.
• Generative AI is dynamic, interacting with new users/data constantly, and risk emerges during operation, not just development.
• Governance should be a runtime property—continuous, embedded, and enforced as part of the system architecture.

“Governance has to exist at runtime… embedded into the system itself through things like policy enforcement, access control… and full traceability.” – Varun Raj [03:26]

• Similar to cloud reliability models: don’t just test, but actively contain and monitor failures as they happen.

“Governance should not only approve AI systems before deployment; it should continuously govern how those systems behave while they operate.” – Varun Raj [04:24]

3. Architectural Separation: Control Plane vs. Data Plane

[05:06]

• Governance controls are often inconsistently scattered across applications, leading to fragmented enforcement.
• A more effective design: separate control plane (makes governance decisions) from data plane (executes AI operations).
• The control plane handles identity verification, policy enforcement, and system-wide rules; the data plane is where AI models run.

“When you centralize governance in a control plane, you can apply least privilege principle continuously and consistently across your entire AI system.” – Varun Raj [06:05]

• This ensures models operate within defined, monitored boundaries, turning previously “black box” models into governed system components.

4. From Model Risk to Systemic Risk Thinking

[07:03]

• Prevailing governance frameworks stress model risk (e.g., bias, fairness, accuracy).
• Most operational failures actually arise from systemic risk—the ways models interact with infrastructure, data, and external tools/APIs.
• Even a great model can cause harm if misintegrated within a less mature system.

“Aircraft safety doesn’t depend on the engine alone. It depends on the entire system—the navigation, the instrumentation, the monitoring and controls. AI systems require the same kind of thinking.” – Varun Raj [07:57]

• Governance must be built into the overall AI system: safety, observability, policy enforcement as core architectural features.

5. Enabling Verifiable, Auditable AI Governance

[08:56]

• Regulators and CISOs now expect tangible evidence, not just assurances of safety/compliance.
• Runtime governance and architectural controls help make organizational AI adoption both verifiable and auditable.
• Ensures transparency for legal, security, and engineering stakeholders.

“Architecting the systems that enforce governance at runtime will help CISOs to ensure that they don’t have the risk that regulators are concerned about.” – Varun Raj [09:34]

---

Notable Quotes & Memorable Moments

• “AI governance needs a similar shift [to cloud reliability]; what this leads to is what I often describe as a runtime governance model for AI system…”
  — Varun Raj [04:13]

• “If a well-trained model [is] placed inside an immature system, it can still produce serious failures.”
  — Varun Raj [08:18]

• “The model is no longer treated as an autonomous black box—it becomes a governed component within the controlled system architecture.”
  — Varun Raj [06:56]

---

Important Segment Timestamps

• [00:29]: Introduction to dynamic AI governance vs. traditional methods
• [03:00]: Explanation of runtime governance and its necessity
• [05:06]: Why separating control and data planes matters
• [07:03]: Differentiating model risk from systemic risk
• [08:56]: Regulatory expectations and enabling auditable AI systems

---

Takeaways for AI Leaders & Practitioners

• Treat governance as a living, always-on property of operational AI—not a static artifact.
• Architect systems so the control plane centrally manages policies, access, and enforcement.
• Evaluate and address systemic risk: the sum of all interactions across the AI ecosystem.
• Build governance into the fabric of the system to provide trustworthy evidence for security, regulatory, and business stakeholders.

---

Transcript

[A]

You're listening to the rsa conference podcast, where the world talks security. Hello listeners. Welcome to this edition of our RSAC podcast series. Thank you for tuning in. I'm Tatiana Sanchez.

[B]

And I'm Casey Zerkis and we are

[A]

your RSAC podcast hosts. Casey, what are we going to discuss today?

[B]

Well, Tatiana, no surprise that AI is front of mind for everyone. And governance around AI, simply put, is just different from traditional governance. Moreover, it needs to be treated differently. And still many organizations continue to approach it as this sort of documentation or compliance exercise. And we know that that's just not working. AI governance is more dynamic and it needs to move beyond just a checklist mindset. So that's why I'm excited. I know you are as well. To be joined today by Varun Raj, who's going to talk to us about how to reframe AI governance through a system design lens. So he's going to offer just a few tips. How organizations can treat governance as a runtime property, how separating AI control planes can create a more secure system, and how teams can shift from a model risk model mindset to a systemic risk. So are you ready to dive in, Tatiana?

[A]

Yes, but before we get started, we want to remind our listeners that here at RSAC we host podcasts twice a month and we encourage you to subscribe, rate and review us on your preferred podcast app so that you can be notified when new tracks are posted. And now we would like to ask our guest to take a quick moment to introduce himself before we dive in. Varun.

[C]

Thank you Tatiana. And thank you Casey. It's great to be here. I am a cloud and engineering executive working on large scale platforms where cloud infrastructure, data systems and machine learning come together to power real production environment. Much of my work focuses on how organizations move generative AI from experimentation into reliable systems that operate safely at scale. What we are learning as an industry is that AI introduces a different category of operational risk than traditional software. With most software, behavior is deterministic. You can review the code and test it. With AI systems, behavior emerges from the interaction between models, data and surrounding platform. So the real governance challenges is no longer simply is the model performing well? The more important question is is the system governing the model behaving safely while I run it in the production? That shift from evaluating models to governing AI systems is where many organizations are focusing right now.

[A]

Thank you Varun. And you mentioned how we need to change our mindsets and look at AI and the behavior and security. Practitioners are seeing that AI risk is behavioral, which is why Governance has to be dynamic rather than static. And organizations shouldn't just treat governance as a policy artifact, but rather as a runtime property. So Varun, can you explain what runtime property is and why does it matter that the system is governed while it runs?

[C]

Absolutely. That's a great question. Because it gets to the heart of why traditional governance approaches don't fully work for AI systems. Historically, governance has often been treated as a static activity. Policies, documentation, approval processes before the deployment. But AI systems are dynamic. They interact with new prompts, new data sets, new users, and sometimes even other AI services. That means risk isn't just introduced during development, it can emerge during operations. So governance has to exist at runtime. It needs to be embedded into the system itself through things like policy enforcement, access control around data retrieval, guardrails around model interaction, and full traceability of what the system is doing. One useful way to think about this is how cloud platform approach reliability. We don't assume reliability just because something passed testing. We build systems that continuously monitor contain failure conditions. AI governance needs a similar shift. What this leads to is what I often describe as a runtime governance model for AI system where policy enforcement, access, control and observability operate continuously as part of the system architecture. Governance should not only approve AI systems before deployment, it should continuously govern how those systems behave while they operate. When governance becomes part of runtime architecture, organizations gain much greater visibility and control over AI behavior.

[B]

So I want to talk about why it is important to have an AI control plane that is separate from the data plane and the applications. So Varun, can you talk to our listeners about not only how, but where they should be enforcing least privilege access?

[C]

Absolutely. And the answer to that question is architecture. And that's why I say this is where the architecture becomes extremely, extremely important. In many early adoption deployments, governance controls are scattered throughout individual applications. Prompts, filters here, safety checks. There's access rules embedded inside different services. The challenge is that governance becomes fragmented and inconsistent. A much more robust approach is to separate the control plane from the data plane, which is a model cloud architects are already very familiar with. The data plane is where the model runs and where interface happens. The control plane is where governance decisions are made. Identity verification, policy valuation, authorization and enforcement of system wide rules. When you centralize governance in a control plane, you can apply least privilege principle continuously and consistently across entire AI system. For example, the control pane can determine which data set a model is allowed to retrieve, which tools an AI agent can invoke, which prompts are permitted and how Outputs are allowed to propagate downstream. This architecture ensures that the model operates within clearly defined boundaries. In other words, the model is no longer treated as an autonomous black box. It becomes a governed component within the controlled system architecture and varun.

[A]

Earlier you talked about shifting a mindset from a model risk thinking to a systemic risk thinking. So can you explain to our listeners what's the difference between the two? And how can organizations mitigate the risk of a good or even a great model being embedded into a not so mature system? What could possibly go wrong?

[C]

There's absolutely I would say this is one of the most important mindset shifts organization need to make as AI becomes embedded into real operational systems. Most governance frameworks today focus on model risk. Things like bias, fairness, the model evaluation, those are important areas, but they only address part of the bigger picture. In production environments, many risks actually emerge from how the model interacts with the surrounding system. A model might perform extremely well in testing, but if that model is connected to sensitive enterprise data, Automated workflows, external APIs or agentic systems, even small error can propagate very quickly. What organizations are really dealing with in those situation is systematic risk. It's no more an operational risk. A helpful analogy is aviation aircraft safety doesn't depend on engine alone. It depends on the entire system. The navigation, the instrumentation, the monitoring and the operational controls. AI system requires the same kind of thinking. If well trained model placed inside an immature system can still produce serious failures. That's why governance has to move beyond evaluating models and towards engineering resilient AI systems where safety, observability and policy enforcement are built directly into the architecture. And that's ultimately what allows organizations to adopt generative AI responsibility at scale.

[B]

What about regulators and CISOs? They want evidence, not only assurances. So when we talk about shifting this mindset for the business, how do we move toward a verifiable and auditable AI model where security, legal and engineering align around what you talk about these architectural controls?

[C]

Absolutely. And the answer to that question is runtime governance model for AI systems. This architecture will allow the organizations to handle the systematic risk which are becoming more and more prominent beyond the model risk. Architecting the systems that enforces governance at runtime will help CISOs to ensure that they don't have the risk that regulators are concerned about allowing organizations to run and operate AI systems in secure and forward looking manner.

[B]

Prune thank you so much for being here today listeners. Thank you for tuning in. Please keep the conversation going on our RSAC membership platform by visiting onersac.com membership and be sure to check onersac.com, for new content posted year round. Until next time.