← Embracing Digital Transformation
Embracing Digital Transformation
Private AI vs Public AI: Governance, Compliance & Cost Control
00:35:46
Loading summary
Transcript55 lines
- [00:00]
Daniel Rodriguez
This is anything that could be really classified under explicit governance requirements and or personally identifiable information. So again, one customer spending an enormous amount of time having to go through data cleansing redaction just to now come up with something that they feel they can comfortably submit to the public AI space.
- [00:24]
Podcast Announcer
Welcome to Embracing Digital Transformation where we explore how people process policy and technology drive effective change. This is Dr. Darren, Chief Enterprise architect, educator, author and most importantly, your host on this episode, Private AI vs. Public AI Governance, Compliance and cost control, with special guest Daniel Rodriguez, Chief AI officer at United Data Technologies.
- [00:56]
Dr. Darren
Daniel, welcome to the show.
- [00:59]
Daniel Rodriguez
Hi Darren. Thank you, thank you for having me.
- [01:02]
Dr. Darren
Hey, I'm really excited about this subject because I'm seeing more and more people and organizations a little fearful of putting all their information up in the public cloud.
- [01:12]
Daniel Rodriguez
Rightfully so.
- [01:14]
Dr. Darren
So we're going to talk about private cloud today and the auctions there. That's some great, incredible things there. But before we dive into that, everyone knows on my show I only have superheroes on the show and every superhero has a background story, an origin story. So Daniel, what's your origin story.
- [01:33]
Daniel Rodriguez
Again? Thank you, Darren for having me on the show. Just a little bit about me. So again, my name is Daniel Rodriguez. I serve as the chief AI officer at United Data Technologies. My career began about 30 years ago in IT I started out as a entry level software developer. Went to school to the most exciting things to me was this idea around personal computers in business. So the platforms I learned on were typically mainframe and mid range systems. But this thing, this PC really started kind of bubbling up in the late 80s, early 90s around something that could have been transformative in the business space.
- [02:13]
Dr. Darren
Wait, wait, wait, I got to stop you there for a second because you said mainframes. So the big question we all have, Daniel, Are you a COBOL programmer?
- [02:22]
Daniel Rodriguez
I am a COBOL programmer. I'm a JCL programmer. I'm a CICS programmer. I also developed RPG on AS4 hundreds. I learned how to write my very first Fortran 77 code on an AT&T3B 5 mini mainframe was my first introduction to Unix. So yeah, that's just a little bit about kind of how I got started in this. But I was really enamored by this thing called the personal computer.
- [02:54]
Dr. Darren
Yeah, my audience really needs to know something. You are almost a unicorn.
- [03:00]
Daniel Rodriguez
Interesting.
- [03:01]
Dr. Darren
There are not very many people left that have actually worked on those boxes because there weren't very many people at the beginning. And now that has, you know, that has gone the way of what you have done too. You moved into PC market, more personal computing.
- [03:18]
Daniel Rodriguez
Yeah, yeah. And I think one of the biggest reasons and what was really exciting at the time was, and you may remember, you know, when you, when you worked on a mainframe system, it was a shared system with shared resources. And as a student at the time, like we have certain Windows, certain schedules with which we had, you know, processor time, the ability to execute jobs. I think one of the funniest things about, about my background was when I got started I really didn't know very much about computers to begin with. I had, you know, taken around with some, you know, Zenith Data Systems and some Commodore 64s and Apple IIS as a kid. But you know, when I, when I started, when I went to college and I started learning about wanting to become software developer and you know, this whole kind of emerging space of it, my very first job was I sat in a little, little office, like a little booth with a window. And for all of the, you know, more senior students, graduate students, my job was every hour on the hour I would go to the maid administration building where the printers were and I would have to collect all of the output printouts, right, because that's how your, that's how your results were delivered. Your output was on dot Matri, you know, high speed printers. So I would go with a little Carter. I would go to the administration building, I'd collect about, you know, four feet sat tall of, of, you know, line paper, printed out job results and then I would have to tear each individual output result and put them in little mailboxes. And then I would sit at the window and students would come up and they would give me their student number and I would hand them their, their job. And then almost always they'd look at the results, they got errors and submission or you know, they didn't properly initialize working storage or whatever it was. And then there was some tears and there was some angst and then they would have to kind of go back on the terminal and kind of resubmit their jobs. But, but that was my very first job in this industry was interestingly enough, collecting printouts every hour on the hour for graduate students who were learning how to be software developers.
- [05:26]
Dr. Darren
Boy, has it changed.
- [05:28]
Daniel Rodriguez
It certainly has. It certainly has.
- [05:30]
Dr. Darren
That is amazing. So let's talk about some major changes, right? Gen hits the the world right? November 30, 2022, a day that should be remembered by everyone. And with ChatGPT3, which now something that really works, that's ready for the whole World everyone jumps on, has fun, they write haiku emails, you write, she's pre installing it. Right. All the fun stuff. But now we start looking. This is real, this is something I can use in my enterprise. But do I want all my enterprise data up in a public thing that they're learning, they're training their models from my interaction with them? Do I want that?
- [06:20]
Daniel Rodriguez
Yeah. These are the big questions that I think our industry is wrestling with. Privacy used to be a really important thing, not only for just individual consumers, where we interacted with technology and our technology vendors and partners of choice. We were very concerned about what you have access to, what they may have visibility to, what kind of data collection is happening. And then as you mentioned, so Genai comes along and the nature of the beast, if you will, is it has an insatiable appetite for information. It's what uniquely distinguishes one vendor's foundation model from others is how relevant, how current, how finely tuned are there specific foundation models? So when we start from a place of insatiable desire for information and systems that have been built to self train and crawl and collect and identify new sources of information, we're already starting at a place where kind of the enemy's at the gate, if you will.
- [07:32]
Dr. Darren
Right.
- [07:33]
Daniel Rodriguez
And they're kind of banging on the door and wanting to get to data and then in some very unique ways are trying to identify what we can use, what we can't use, what's personally identifiable information. But this is a real gray area right now. And more importantly, what we're finding is the space is really fraught with a lot of challenges around what's considered fair use. So if we look at what's happening around all of the legal entanglements that's happening around a lot of the major suppliers of generative AI, public AI platforms, nearly all of them are struggling with their technologies, scooping up information, training foundation models to then only become made aware of by the original content creator or intellectual property owner that they may have infringed upon the use of their knowledge.
- [08:44]
Dr. Darren
Yeah. Do we look the other way on this? Because the benefit so outweighed the, the rights, the copyright rights there that for the public good we.
- [08:57]
Daniel Rodriguez
The other way.
- [08:58]
Dr. Darren
That's kind of what seems like happened.
- [09:01]
Daniel Rodriguez
Yeah, well, you know, it's interesting. I. So there's, there's what I like to kind of call, you know, facts, things that we know in the industry and then there's, you know, personal opinions, feelings, et cetera. Right. So the facts are this. There are dozens and dozens axios Did a great job of visualizing this just a few weeks ago. Of all of the legal entanglements where there are lawsuits or litigation in place, probably the most publicly circulated one as of late is the one between the New York Times and OpenAI that has caught Microsoft up as well in the use of some pirated materials that were made available in the public space that were drawn up in their training. And the New York Times has kind of a legitimate position that said this is our intellectual property and you're not licensed to use this. And so I think when we think of what's happening around the issues around what's considered fair use and what should be used, how in this way also begins to borderline of what I like to just kind of my opinion on things. And, and I believe the hyperscalers, the public AI folks, they already have litigation and settlement costs baked into the calculus. Like they know, because these things are insatiable desire for information. They're going to collect up as much as they can, they're going to use as much as they can. And when they do kind of borderline cross into this, what is considered fair use, not fair use, maybe we shouldn't have used it. We don't know the original source of the information. And if there's any intellectual property or intellectual rights around that, I think a lot of that opportunity cost is baked in because the prize at the end. Right. So AGI, the value and the opportunity is so great. The, the opportunity costs along the way of having to settle some of these, put some agreements in place, like they can't slow down and have to try to solve every single one of these, you know, potential litigation issues. So I think they're just hard charging straight through this industry.
- [11:13]
Dr. Darren
Yeah.
- [11:13]
Daniel Rodriguez
And then just like the cost of this is just again, it's baked into the calculus.
- [11:18]
Dr. Darren
So, so that, that's an interesting thing because they don't charge ahead, their competition will. Which mean I, I should say their competition meaning China, because China doesn't have the same intellectual property laws we do in the West. And even in their culture they don't. They. It's a cultural thing. How can you own an idea that just doesn't calculate in their culture? So yeah, we try to do something, but we've kind of. And so how do I as an individual now prevent, prevent my public data or my personal data being pushed up into these big eight, nine, what options do I have?
- [12:05]
Daniel Rodriguez
Yeah. And you know, this is the genesis and if not the hypothesis of the work that we've been doing and what we're focused on around private AI platforms. So we look at the marketplace obviously as much as the industry does. There's what's available in the consumer marketplace and nearly every single generative AI solution and or platform today really kind of started out in this consumer space and this is where we're directly familiar with things like Copilot and ChatGPT for OpenAI and Google, Gemini, Anthropic and the likes, really targeted towards the consumer marketplace. That very traditional per user per month and then tiered offerings and capabilities. So starting in consumer and then they're racing into the enterprise, you know, quickly trying to adopt, integrate and adhere as much of their technology as possible directly into the enterprise. Now unfortunately, because the starting point was and like I said, nature of the beast, insatiable desire for information. Starting in the consumer side, some of the controls if you will, around, hey, don't use my prompts, don't use my information, don't use my data for or foundation training do not even begin to rise to the requirement and the responsibility of organizations are looking at as it relates to governance and compliance and then more importantly how they may be, you know, managed from a regulatory compliance space. So in healthcare, around hipaa, in education, around ferpa, just to name a couple, like those controls don't even begin to rise to the capabilities and or requirements necessary to help organizations kind of not only meet their own internal compliance requirements, but what they're also being, you know, managed to adhere to.
- [13:58]
Dr. Darren
Well and this is a problem for them, right, because now they're being left behind in this AI revolution that's going on, right where they, they can't take advantage of these large language models if they're concerned about data privacy. Right. If they're regulated, I can't use this stuff that's very valuable in helping me streamline my processes and operational efficiency. All I can't take advantage of.
- [14:27]
Daniel Rodriguez
Yeah. And what's really wild about all of this, I just met with a client just a couple of days ago and they really, really want to use this technology and they're trying to put this into the hands of their users. But these end users are now being trained to have to do a tremendous amount of either reduction and, or data cleansing before they could even begin to submit a kind of initial work product to begin to kind of generate some type of insights around and this is predominantly around obviously what we classify or consider as semi structured or unstructured data assets. Very few if any have really begun to target a lot of their structured data because of the challenge of specifically this is financials, this is customer data, this is anything that could be really classified under explicit governance requirements and or personally identifiable information. So again, one customer spending an enormous amount of time having to go through data cleansing, redaction just to now come up with something that they feel they can comfortably submit to the public AI space.
- [15:48]
Dr. Darren
Let's talk about Genai because we're dancing around it a little bit. So let's talk about what it does and doesn't do that, solving this problem.
- [15:58]
Daniel Rodriguez
Yeah. So what is private AI platforms? Well, private AI platforms, when we think in terms of infrastructure requirements, we think about the technology itself. You know, we've spent as an industry the last 10 years or more building multi tenant cloud native applications. Right. So the economics that customers were gravitating towards what solution providers were building large ISVs was this idea that we have this thing now called the cloud. And in the cloud we have some kind of basic rules. Number one, it's lots and lots and lots of commodity hardware. Number two, we are continuing along this trajectory of hyper consolidation. So going back. Wow. Now, okay, so let's go back. A little more than 30 years ago we started our conversation around the mainframe and it's interestingly enough some of these things start to come full circle. So we think about the mainframe and we think about the fact that this was proprietary hardware platforms, very expensive. So the cost for compute was very high. And then open systems began to challenge those underlying economics. And then we saw scale out compute and client server and client server then led to consolidation in the data center and virtualization and then hyper consolidation in the cloud. And so the economics were continuing to be built around a financial model that said there's this declining cost of compute, lots and lots of, you know, just commodity hardware capabilities, CPU memory and storage. And now the cloud represents an infinite, for all intents and purposes is infinite access to unlimited commodity resources that we could even further hyper consolidate into. And then along comes Genai And Genai is 180 degrees of those economics. It's no longer commodity hardware, it's proprietary hardware. Again, kind of similarly to the mainframe, really expensive. When we look at GPUs and high core count, CPUs, large amounts of unified memory to be able to load and deliver these very high parameter single precision or half precision language models. And so we need a lot of resources. These are pretty expensive resources. And so therefore what we're beginning to see is that there is this inherent challenge around what are the compute domains that I need to activate in order to be able to deliver generative AI through this tokenomics, this kind of new economic model that's very different than cloud economics in a way that's sustainable and manageable for a large organization. So private AI at its kind of original intent is well it's not multi tenant and it's not a SaaS platform, it's a distribution. So customers have the ability to deploy in the cloud if they'd like or they could deploy at a fully air gapped data center with absolutely no connectivity to the outside world. So let's say for example, a customer has this very, very sensitive data sets that they're working with and they want to absolutely ensure that they can walk into a data center and say the complete and total footprint of this private AI platform sits in these racks on these intel servers. And there are no language models, there's no prompt injection, there's no vector databases or rags or anything that contributes to the delivery of generative AI sits outside of that. It is completely air gapped and sits inside of this environment. So we start with freedom of choice. Where do you want to put this platform and what specific outcomes are you trying to deliver around that? Around governance and compliance. And then the second key tenant of private AI platforms is being able to land language models where they make sense based upon your consumption requirements.
- [20:02]
Dr. Darren
Well I'm glad you brought that up because well what you mentioned before, open source or there was disruption. Right. The open source of these large language models are disrupting the public gen AI and furthering along a lot further. So I can actually stand up my own gen AI or large language model in my own environment. But you mentioned all the other things around it as well. It's not just standing up large language model, there's a lot of other moving components around Modern or the latest gen AI. That open source concept has actually made it possible. So I can do this myself, which is great.
- [20:49]
Daniel Rodriguez
Yeah, absolutely. Interestingly enough. So when we look at again what I'll just kind of define are these four compute domains today. OpenAI, Microsoft through Azure, OpenAI as a Service, Google through Vertex, the Vertex Cloud or Gemini. They'll provide you APIs and you can start to build and integrate applications and have access to the most powerful foundation models available.
- [21:12]
Dr. Darren
Right?
- [21:12]
Daniel Rodriguez
The most highly trained, most relevant, the most accurate foundation models and there's a place in those for a lot of organizations. There are some specific outcomes that we may want to be able to deploy and deliver those types of models. So we'll call that platform as a service it's the most expensive compute domain because at scale based upon what I like to refer to as the holy trilogy of cost associated with generative AI. It's your input tokens, it's your output tokens and it's your textual emittance. Right. So that's where there's actual costs that's associated with the consumption of those technologies in particular. And so when we use these platform as a service, these API accessible foundation models at scale for a lot of organizations, this becomes very expensive. Number two, it's also pretty unpredictable. So you know, a lot of industries they require, you know, some predictability around budgets that they put into the budget. We're going to spend 100,000, 200,000 on generative AI subscription through the Azure cloud this month and then suddenly they get the surprise next month and it's now.
- [22:17]
Dr. Darren
Well we saw the same thing with cloud computing repeating what we've already done.
- [22:37]
Daniel Rodriguez
Yeah, so as well. So what's interesting is that though necessary, it's not required for every generative AI outcome. So then we turn our attention to the second compute domain and we're going to call this infrastructure ASARCS. So again we spent 10, 15 years building clouds and hyper consolidating and getting really comfortable with this idea that I can stand up infrastructure really quickly and grab some intel compute in the cloud and I can grab some GPU and I can throw this together and deliver an outcome in a matter of minutes or hours as opposed to, you know, weeks or months to provision in the data center. And so the ability to now run things like VLLM and open source models gives me the opportunity to jump into a marketplace of language models like hugging face, download some open source models and then deploy those directly into the cloud. One particular model of interest that we've been working with lately is a model called medgemma. And so this is a model, it's available from Google through Vertex. So platform as a service, really expensive by comparison when we think about deploying directly into infrastructure Asura so rough and tough you run that particular and this is a Gemma 3 based model, finely trained or finely tuned, excuse me, on a medical corpus of knowledge. So think of lab Results, x rays, MRIs, lots and lots of medical information, medical knowledge. So it can be your expert radiologist, hematologists, oncologists, and now submit multimodal types of assets so images, videos.
- [24:22]
Dr. Darren
But there's a problem that doing that with public, with the public gen AI. The problem is HIPAA is HIPAA.
- [24:28]
Daniel Rodriguez
Right. So if I gave that to ChatGPT there are concerns about like how is my information going to be used for an organization that's responsible to be compliant to hipaa, how can they successfully use customer information, patient information in a way, and then also make sure that they're 100% compliant around data sharing requirements. So being able to deploy these open source models like a mengeema, not through the vertex cloud, but now specifically an infrastructure service, put it in your gcp, put it on Azure, put it on AWS standalone vm. Now you have controls where the prompts are, you know how prompts are submitted, all the textual embeddings, any documents that are being uploaded, all of that is delivered through your cloud tenant. And that's great. The underlying economics now become very predictable. I can only deliver a certain amount of sustained tokens per second through infrastructure as a service. It's not unlimited like it is for example when I'm consuming it through API, through the, through the public AI providers. So I have a finite amount depending upon how I engineer this in the cloud. But the cost now again rough and tough like $25 an hour to run this one model IT infrastructure as a service. So we'll call that the second compute domain. We now begin to inject this kind of full privacy capabilities into your tenant. I can stand up and tear down infrastructure quickly if I needed to for specific short term outcomes based upon a generative AI requirement in the business. But the two under exploited compute domains are the ones that we're probably most excited about. The third is back in the data center, right? So we have spent 10 to 15 years moving workloads out of the data center and up into the cloud. And the economics of cloud supported that. Going back to and referring to the economics of AI being 180 degrees of that, complete opposite kind of economics of that. It's beginning to make a lot of sense for customers to begin to reinvest in the data center and then beginning to land these new open source language models in something that they can acquire and depreciate over time because it's a finite amount of capacity, it is proprietary, it is expensive, and what we find is we can almost cut in half the cost of delivery of those foundation models to when we deliver them in the data center. So we go from about $25 an hour to somewhere just under $12 an hour when we're operating inside of the data center through kind of a shared services model. So there is a new renaissance opportunity in the data center. All of the, you know, our silicon partners like intel are really excited about what this means for customers. And more importantly, as we talked about, not every requirement for generative AI requires these 400 billion parameter large language mod.
- [27:34]
Dr. Darren
Exactly. If I wanted to just summarize my own documentation in English, I don't need a 4 billion or trillion parameter model.
- [27:56]
Daniel Rodriguez
These crazy models, right? I can deploy a 4 billion, a 20, 27 billion 40 billion parameter model on infrastructure in the data center and get really, really good results. And then especially if I'm focused on specific outcomes that I'm using finely tuned models, then I can really narrow my focus. Again, in the Med Gemma example, I can use a 27 billion parameter text only model to look at in medical records and summarize and look for insights and create patterns. And it's all based upon this gum trained medical corpus. Or I could use a 4 billion parameter version of the Med Gemma model and use multimodal and have it analyze, you know, x rays and EKGs and you know, MRIs and correlate that with textual information and begin to help create some, you know, really fine tuned medical expertise through our language models. And there isn't a requirement to have to run up this solely through platform as a service because we can deliver infrastructure in the data center to deliver that. So now this customer has a private AI platform, they're hosting their models in the data center and they can 100% ensure that they're meeting HIPAA compliance around the risk associated with that. That's awesome.
- [29:06]
Dr. Darren
Now the fourth.
- [29:07]
Daniel Rodriguez
Yeah, that's our third one.
- [29:08]
Dr. Darren
What's the fourth one?
- [29:10]
Daniel Rodriguez
Okay, I'm really excited about the fourth one.
- [29:12]
Dr. Darren
We're going back to the PC.
- [29:15]
Daniel Rodriguez
Welcome back to client side computing. Right, so, so here's, here's what's exciting about that and I'll share with you some information and some interesting statistics that are starting to validate this thing that we call the AI PC. These AI powered PCs. We're starting to see some really interesting industry data that's now validating that the AI PC is quickly becoming the standard sec, not only in consumer, but also on the commercial side of the supply chain and if you will, the kind of consumption chain of technology today. But what is an AI PC? An AI PC is not unlike a traditional PC, a laptop, a desktop, but it does have kind of a uniquely distinguished element around it and that's something that we refer to as an NPU. Now we're all very familiar with GPUs, these graphical processing units, and we've been building and deploying these for a long time. And they power our gaming experiences, they deliver high frame rates and they're, they're really engineered to offload a certain type of compute from the CPU in a way that helps preserve, well, creates greater performance, but in some cases can help preserve better battery life in our mobile devices, deliver overall better experiences. When we begin to uniquely provision a workload that's built for a uniquely designed chipset to deliver a specific experience or capability. So along comes this thing called an npu. It's a neural processing unit. It's another piece of discrete silicon now inside of these AI PCs, and these are built for generative AI. So specifically, when we think about how language models work and these kind of large graft models, highly complex algorithmic and mathematical work, to begin to connect and correlate the understanding and the meaning of words and language and how that all comes together to deliver, deliver a response. In short, it's a piece of silicon that is really uniquely built to deliver this at scale. We measure the performance of these and refer to as a metric called tops, trillions of operations per second. And today In a standard AI PC, we have the ability to deliver with just an NPU somewhere between 20 to 70 or so tops out at the edge. So what does that mean? This fourth compute domain begins to represent more tops for large customers out at the edge than they'll ever want to pay for in the data center or subscribe to in the cloud. And becomes incredibly important that as organizations are looking at deploying generative AI, that they have the opportunity to push inferencing, because that's ultimately what's happening, inferencing as much as possible out to the edge. So in the same vein that we're looking to deploy open source models, what types of models can we deploy locally, take advantage of that npu, so all of that inferencing capability out at the edge, and then deliver an economic outcome for customers to where the cost for inferencing is already baked into the cost of the device. I no longer need to build in the data center, expand in the data center at the scale that I needed to to be able to deal with my inferencing requirements. And I certainly don't need to subscribe to or provision cloud resources in a way that presents in some cases just an unmanageable out of control costs around inferencing in the cloud. So if we activate all four successfully, customers now have choice how do I provision it? What outcomes am I powering by? What agents am I assigning to these four different kind of compute domains. And when we look at industries like education or healthcare, where we have a high density of endpoints, it only makes sense for customers to be able to activate edge inferencing.
- [33:11]
Dr. Darren
All right, so we're running out of time, Daniel. So we got to figure, figure something out.
- [33:17]
Daniel Rodriguez
We're just getting warmed up there. I know.
- [33:19]
Dr. Darren
We're just getting warmed up. We probably need to have you come back on. But if people want to find out more how you know how to go about leveraging all four domains, how did they reach out to you and your company? Udt. Right. How do they reach out to you to find out more about all of this? And where do they go to find out how to leverage all this great new stuff that's out there instead of always pointing to ChatGPT or Gemini or.
- [33:50]
Daniel Rodriguez
How do I get started? Yeah. Great. Well, again, thank you again for this opportunity. You can always reach out to me and learn more at our website, www.udtonline.com. we're a full service IT solutions provider making significant investments in this AI domain. And so there's an opportunity to download some white papers. We have a bunch of one pagers that are industry aligned. There's always some links to be able to click out to kind of contact us directly. And we're happy to share kind of more around this specific area around private AI platform. But more importantly, these architectural enablers that are beginning to activate these four compute domains for customers in their private AI endeavors.
- [34:37]
Dr. Darren
This is awesome because we're not leaving industry behind. And regardless of what all these public gen AI CEOs are saying that everyone's job is going to be replaced, that's total garbage. We need people out there and this is breaking down some of those barriers that they're trying to establish. So I, I love, I love this stuff. Daniel, this, this has been great. Thanks so much for coming on the show.
- [35:05]
Daniel Rodriguez
Thank you, Darren. Thanks for having me.
- [35:11]
Podcast Announcer
Thanks for listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community@patreon.com embracingdigital where we share bonus content and you can always connect with other change makers like yourself. You can always find more resources@embracingdigital.org until next time, keep embracing the digital Transformation.