Podcast Summary

Podcast: Embracing Digital Transformation
Episode: Private AI vs Public AI: Governance, Compliance & Cost Control
Host: Dr. Darren Pulsipher
Guest: Daniel Rodriguez, Chief AI Officer at United Data Technologies
Date: November 6, 2025

Overview

In this episode, Dr. Darren Pulsipher and guest Daniel Rodriguez dive deep into the growing divide between private and public AI, focusing on governance, compliance, and cost control in public sector digital transformation. The discussion dissects risks with pushing sensitive data into public AI platforms, emerging architectures for secure and compliant private AI, and innovative compute solutions bridging the value and privacy gap for organizations in heavily regulated environments.

Key Discussion Points and Insights

1. Daniel Rodriguez’s Background and the Evolution of IT

• Origin Story: Daniel’s 30-year IT career began as a software developer working with mainframes, PCs, and now AI.
  • Memorable anecdote about handing out job printouts to students (04:37).
  • “There are not very many people left that have actually worked on those boxes ... you are almost a unicorn.” — Dr. Darren (03:01)
• Technological Shift: From the restrictive, shared nature of mainframes to the democratization of computing via PCs — and now, consolidation via cloud and AI.

2. The Privacy Risks and Legal Quagmire of Public AI

• Public Cloud Hesitance: Enterprises are increasingly wary of exposing sensitive or regulated data to public AI tools (06:20).
• Insatiable Appetite of Generative AI: Public models need vast data, often scraping anything available — creating liability and fair use questions (07:00).
  • “The nature of the beast... is it has an insatiable appetite for information.” — Daniel (06:26)
• Legal Battles: Ongoing lawsuits (e.g., NY Times vs. OpenAI) over data used in model training; providers factor litigation costs into their strategies (09:01).
  • “The hyperscalers, the public AI folks, they already have litigation and settlement costs baked into the calculus.” — Daniel (10:25)
• Global Asymmetry: U.S./Western tech firms face different IP standards than, e.g., China, which accelerates the need for innovation without overexposure (11:17).

3. Governance, Compliance, and Challenges for Regulated Sectors

• Regulatory Barriers: Sectors governed by HIPAA, FERPA, and similar cannot use public AI in the same way due to strict compliance needs (13:00).
• Operational Friction: Organizations must cleanse and redact data before using public AI — highly inefficient (14:26).
  • “One customer spending an enormous amount of time ... just to now come up with something that they feel they can comfortably submit to the public AI space.” — Daniel (15:41)

4. What is Private AI? The Four Domains of AI Compute

Daniel introduces a four-pronged framework for deploying generative AI securely and cost-effectively:

1. Public Cloud / Platform-as-a-Service (PaaS)

• Pros: Access to powerful, up-to-date models/APIs from OpenAI, Azure, Google, etc.
• Cons: Expensive, cost difficult to predict at scale, few data governance guarantees (21:11).
• Quote: “At scale... this becomes very expensive. Number two, it’s also pretty unpredictable.” — Daniel (21:22)

2. Infrastructure-as-a-Service (IaaS) / Private Cloud

• Deployment: Running open-source models (e.g., via Hugging Face) within the organization’s secure cloud tenancy for better control.
• Pros: Predictable costs, privacy, fast provisioning (24:28).
• Example: Running “MedGemma” (medical AI model) for healthcare data, maintaining HIPAA compliance.
  • Quote: “Now you have controls where the prompts are, how prompts are submitted... all of that is delivered through your cloud tenant.” — Daniel (24:44)
• Cost Note: ~$25/hour for specific workloads (26:17).

3. On-Premises Data Center

• Shift Back: The economics of AI (expensive, specialized hardware) are prompting a renaissance in local data center investment.
• Cost Efficiency: On-prem solutions can potentially halve costs compared to cloud (from $25/hr to ~$12/hr; 27:14).
• Use Cases: Finely-tuned and specialized models (e.g., summarizing internal medical records) without risking data leakage.
  • Quote: “We can almost cut in half the cost of delivery of those foundation models... in the data center.” — Daniel (27:25)

4. Edge / Client-Side Computing (AI PCs)

• AI PCs: New class of end-user devices equipped with Neural Processing Units (NPUs), offering dedicated on-device AI inference.
• Massive Potential: Offloading inference to endpoints is highly scalable and cost-effective; especially appealing in healthcare and education, where thousands of endpoints exist (29:15–32:20).
• Quote: “This fourth compute domain begins to represent more TOPS for large customers out at the edge than they’ll ever want to pay for in the data center or subscribe to in the cloud.” — Daniel (32:05)

5. The Future: Architectural Flexibility and Keeping Industries Competitive

• Empowering organizations to securely leverage generative AI (across any of the four domains), without sacrificing compliance nor surrendering budgets to the unpredictable costs (33:11).
• Private AI as an enabler, rather than a barrier, to transformation — especially for sectors traditionally left behind by big public AI leaps.
  • Quote: “We’re not leaving industry behind. Regardless of what all these public gen AI CEOs are saying ... we need people out there and this [private AI] is breaking down some of those barriers.” — Dr. Darren (34:37)

Notable Quotes & Memorable Moments

• On Mainframe Nostalgia & Modern AI:
  “You are almost a unicorn. There are not very many people left that have actually worked on those boxes...” — Dr. Darren (03:01)

• On Data Risk in AI Models:
  "The nature of the beast, if you will, is it has an insatiable appetite for information." — Daniel Rodriguez (06:26)

• On Legal Risk, IP, and Public AI:
  "The hyperscalers ... already have litigation and settlement costs baked into the calculus." — Daniel Rodriguez (10:25)

• On Private AI as the Solution:
  “Customers have the ability to deploy in the cloud if they’d like or they could deploy at a fully air gapped data center... complete and total footprint ... sits inside of this environment.” — Daniel Rodriguez (18:27)

• On the PC’s AI Comeback:
  “Welcome back to client side computing... The AI PC is quickly becoming the standard...” — Daniel Rodriguez (29:15)

• On Keeping People at the Heart of Transformation:
  “Regardless of what all these public gen AI CEOs are saying that everyone’s job is going to be replaced, that’s total garbage. We need people out there and this is breaking down some of those barriers...” — Dr. Darren (34:37)

Key Timestamps for Important Segments

• Daniel’s IT “Origin Story”: 01:32–05:26
• Rise of Generative AI & Privacy Fears: 05:29–07:32
• Legal and Ethical Challenges with Public AI: 07:32–11:12
• Options for Controlled AI Use & Private AI Emergence: 12:05–15:57
• Comprehensive Breakdown of Four Private AI Domains: 15:57–33:11
  • Public Cloud (PaaS): 20:48–22:16
  • Private Cloud/IaaS: 22:36–26:17
  • On-Prem Data Center: 26:17–29:06
  • Edge/PC: 29:09–33:11
• Closing Reflections & Call to Action: 33:11–34:37

Conclusion

Takeaway:
Private AI is rapidly emerging as a critical strategy for regulated industries to gain the advantages of generative AI while maintaining strict control over sensitive data, compliance, and cost. By leveraging flexible compute domains — cloud, private cloud, on-premises, and edge — organizations can balance innovation with governance and scalability.

Contact/Next Steps:
For more resources on adopting private AI architectures, Daniel Rodriguez points to udtonline.com.

Episode tone:
Practical, engaging, insightful — a balance of technical depth and real-world relevance for IT leaders navigating the digital transformation landscape.