← Embracing Digital Transformation
Loading summary
Transcript126 lines
- [00:00]
A
Without an mpu, you wouldn't really be able to efficiently run those, you know, and I think that those are sort of some of the sort of future ways that people are going to be using these sort of tools. I think as we move into.
- [00:18]
B
You really think, having that, like that personal generative AI sitting. Maybe we should call it genie. A personal genie.
- [00:27]
A
A personal gene.
- [00:28]
B
Yeah, There you go, Janae. Welcome to Embracing Digital Transformation, where we explore how people process policy and technology drive effective change. This is Dr. Darren, Chief Enterprise architect, educator, author, and most importantly, your host on this episode, personal AI and privacy. First, how NPUs and AI PCs disrupt public generative AI, with special guest Dan Salinas, chief operating officer from Lakeside Software. Dan, welcome to the show.
- [01:07]
A
Hey, Darren, how you doing? Good to be here.
- [01:10]
B
I'm doing pretty good. It's been a busy morning already here in California, where I'm at, and today we're going to talk about a new platform of computing that's just. It's really incredible what you're able to do with it, and we're going to talk about that and the impacts on that. But before we do that, Dan, everyone that listens to my show knows that I only have superheroes on the show because I would never have anything less. And every superhero has a background story or an origin story. So, Dan, what's your origin story?
- [01:46]
A
Oh, man, so many there. But I guess the thing that is the most maybe unique thing about myself has nothing to do with myself in some way. So my fantasy football team name is Quadfather, and the reason it is Quadfather is because I actually have quadruplets. Well, that's.
- [02:08]
B
You. That's awesome.
- [02:10]
A
Yeah.
- [02:11]
B
Are they identical quadruplets or.
- [02:15]
A
So paternal. Two boys and fraternal. Two boys and two girls, 19 years old. They just. Just went off to college, actually, so my wife and I are empty. Nesting.
- [02:26]
B
Oh, my goodness. What a. What a journey. That must have been an incredible journey, right? I mean, I. I don't remember.
- [02:35]
A
I don't remember the first four years of their lives. So there's pictures, but I guess I.
- [02:41]
B
Was there for four years.
- [02:44]
A
Well, I mean, you know, with kids, you don't sleep anyway, so I guess. I guess, you know, there's only so much sleep you can lose, I suppose, but.
- [02:54]
B
Oh, damn, that. That is. That is really amazing. I have 10 kids, and I always tell people the. The odd number of kids are the hardest, meaning the first one. And because you're, you know, it's life changing. Having a child is life changing. And then the Third one because it's the first time you're outnumbered, but when you have quadruplets you get all that all at once. That's, that's.
- [03:18]
A
Well, I mean we struggled with the first one, but we had it figured out by the fourth one. I mean we really, you know, four minutes in between was plenty of time, you know. Well, 10 kids though. That's a, that, that, that, that I can't even, you know, I don't. After.
- [03:37]
B
Well, I can't even imagine.
- [03:38]
A
I can't even imagine anymore.
- [03:40]
B
Oh man, that's so awesome. So with all that great knowledge of playing defense and, and coordinating, how does that apply to, to what you do in, in your day to day life?
- [03:58]
A
Well, I suppose learning to balance time is an important thing. You know, it's is as, as they've, they've, they've grown up, you realize, you know how valuable, how valuable that is and you want to work on things that are impactful when you're, you're in your job world. I guess, I guess I take it from there. I mean, I've really been lucky that I've had a bunch of organizations I work with that have been very supportive of it, you know, so.
- [04:35]
B
Well, that, that's awesome.
- [04:37]
A
So.
- [04:37]
B
Hey, let, let's dive, let's dive in into the topic today. I mean, AI PCs were unleashed on the world about two years ago kind of, right? So we've heard the term. What in the world does it mean?
- [04:53]
A
Well, I mean, I think it's, it's basically being able to use the compute power of a PC at the edge to run these advanced AI models, right? So instead of spending all your money in the, in the cloud, you can leverage the power of a distributed device to, to get use, use the, use the AI engine. I mean it could be as simple as, you know, leveraging the. When you're on a zoom and you're running these funky backgrounds that need, you know, good power to, to run them, so you can, you, you can use it that way or you can run local LLMs. And some of the, some of the software vendors are, are starting to, starting to take advantage of that. Systrack being one of them. Actually we run, we analyze our data using npus. So.
- [05:53]
B
All right, so you mentioned npu. What? This is a new term that a lot of people have never heard. I know what it is because I work at Intel, I know what npus are. But can you explain this new. Because most people are used to, hey, I buy a laptop, it has a CPU in it, a Central processing unit. I have maybe a gpu, a graphics processor. So what's this concept you mentioned, npu. What's that all about?
- [06:19]
A
Well, I guess it stands for neural processing unit. Really. So it's really focused on performing AI and machine learning tasks. I think GPUs can also do AI tasks, but NPUs being neural processing units, they can perform, they're more tailored to perform AI tasks. So they're more specialized than GPUs. They're more efficient.
- [06:46]
B
Well, I like, I like the word efficient because they're tailored specifically for that type of workload. So we can, we can decrease the power dramatically by doing this.
- [06:57]
A
Right, right, right. Yeah. GPUs take a lot of power. So you could tell with the fan spinning in your laptop when you're, when you're using your GPU versus your mpu.
- [07:08]
B
Yeah, the fan is actually on when.
- [07:09]
A
You'Re using the gpu, Right. It's spinning like crazy. Yeah.
- [07:14]
B
So what kind of use cases do you do you see this new processing unit kind of open up to the world besides the, I mean, you mentioned, oh, I can run a large language model on my laptop now. So what? Or I can do better blurring on my zoom. Yeah, I have that already. That's cool. But what is this bringing me that's so different?
- [07:42]
A
Well, I mean, I think, you think in terms of like where we're going with using personal LLMs, like do you want your data to be in the cloud, shared with all the providers that are basically learning for your data, or do you want to have something private to yourself? A. From a security perspective, without an mpu, you wouldn't really be able to efficiently run those, you know, and, and I think that those are sort of some of the, the, the sort of future ways that people are going to be using these sort of tools. I think as we move into.
- [08:24]
B
You really think having that, like that personal generative A.I. sitting. Maybe, maybe we should call it genie. A personal genie.
- [08:33]
A
A Personal Genie. Yeah.
- [08:34]
B
Or. Jana. Jana, Jana. There you go.
- [08:36]
A
Jana.
- [08:38]
B
A personal gen AI running on your, on your laptop means that your data is not leaving your laptop. That's. That's huge.
- [08:44]
A
Yes.
- [08:57]
B
I mean that's, that's going to cut into OpenAI and, and Gemini and all the things. Can it do everything that those models can do? Running these large or small language models on my laptop, can they do all the same things that the big ones can do?
- [09:13]
A
Well, I think, I think if you look at like when you're using Chat GPT, you'll notice that when it's doing its reasoning, a lot of that is utilizing, doing neural network processing that can be done at the edge. So you can combine. So we're getting data from the cloud such that then that processing, when it's doing that reasoning and that thinking can happen on your AI PC. So that enables you to operate more efficiently and securely so that data doesn't actually leave your personal device.
- [09:51]
B
Oh man, this is opening up a whole bunch of interesting use cases in my crazy head of mine of things that I can do now, right. Because it's sitting on my laptop. Instead of pushing data up to the edge, there's lots of different things I could do both in a personal thing. And I'm also thinking military use cases, even like a FedEx driver. Can you imagine a FedEx driver having a large language model sitting on his mobile device right there? You know, that would be super cool.
- [10:26]
A
Yeah, yeah. And I think when we see where AI goes, we're sort of in the what can it do phase, but we haven't really thought about what can it do economically. So like, even in our own product, as we analyze where we want to do the AI processing, doing it in the cloud is expensive and depending on, you know, how much you, how big the session is and how much you feed into the session. So being able to do it on the edge, you're going to be able to do the processing much cheaper. We sort of saw this with, with virtualization, you know, the first time around. Like when we look to virtualize a desktop, for instance, which is basically running a desktop in the cloud.
- [11:19]
B
A lot.
- [11:19]
A
Of the economics of that didn't make a lot of sense because like the processing power of a desktop, you know, a MIP on a desktop, if I can use that term, millies in a stretcher per second ages me a little bit. You know, the processing power of that, the cost of that compared to a MIP in the cloud is like a tenth. You know, the cost of the memory is, is, is cheaper on a, on a laptop. So at the edge you can, you can, you can do more compute at a cheaper rate than you can in the cloud. And I think when we start to look at what AI costs, once we get past the, the, you know, it starts to normalize as technology that people are using, people are going to look to optimize the cost us to the use.
- [12:05]
B
We're already seeing a huge impact on energy with all these AI data centers being built like Stargate. I mean, Microsoft bought Three Mile Island. Why did Microsoft buy a nuclear Power plant.
- [12:21]
A
Exactly.
- [12:22]
B
Well, because they could put a data center on Three Mile Island. Right. And not have to pay the energy costs. That's crazy. Whoever would have thought that Microsoft would own a power plant? It just doesn't make sense.
- [12:36]
A
So, yeah, I mean, it's certainly a little back to the future. Right. You know, back when we were building big data centers for mainframes back in the day, and all that processing power was in the data center. And I think, I think, you know, it is cyclical. Right. I think people started to realize, like, maybe that's not the smartest model in the world because all of that consolidated power is expensive. So maybe client server computing made some sense, you know.
- [13:11]
B
So do you think we're headed back? We're headed on a swing back into.
- [13:15]
A
I think we are fatter clients, I think so. I just, you know, there isn't enough electricity, you know, power and data center capability to support where computing models are going. So you're going to have to be very selective on what workload you want to run in the cloud and then you want to leverage, you know, the power of all these distributed devices that we have and. Yeah, so they're just going to run out of capability. Right, Right.
- [13:48]
B
Yeah. Now, this is really interesting because now that I think about it, if. If I have an organization that has a thousand employees and I give them all an aipc, which the cost on an AIPC is not much more than a normal laptop.
- [14:03]
A
Right.
- [14:03]
B
We're only talking, you know, a couple hundred dollars, maybe $300. Then, wow, all of a sudden I have a thousand NPUs out there running with GPUs as well. That means I could, I could easily have a thousand or two thousand large language models running close to where the data is being generated. That's pretty impressive. That's. But how in the world do I manage something like that?
- [14:33]
A
Yeah, there's gonna, there's gonna need to be tools built to, to do that. I think people are working on that, system management tools, but, you know, it's conquered ground a little bit. I mean, if you remember the old SETI at home. Oh, yeah, yeah. You know, like I did SETI at home, like, you know, and it was leveraging extra CPU power to conquer big, big models. And I think, you know, I think we're going to see tools and software built to chop up LLMs and spin them out and have them. You know, I think that's going to happen.
- [15:06]
B
That would be, that would be pretty slick if I had a collaborative LLM cohort or HIVE if we're just my organization where they, you know, hey, my laptop's sitting there, you know, probably plugged in.
- [15:22]
A
Yeah.
- [15:23]
B
Why not use, why not use the cycles to help the corporation? That would be, that'd be really slick.
- [15:31]
A
Yeah, I mean, I think we were starting to see people do that and I, and I, you know, just because the capacity won't be there in the cloud. So I think these are going to be good use cases for MPUs on top of, you know, just applications that'll be able to take advantage of them, like, you know, real time translation, you know, all sorts of different use cases that MPUs really help with. Right. So like, you and I are, you know, know, having a call where we're both talking in English, but I can see in the very near future I could be talking English, you could be talking in a totally different language, or maybe I'd be talking in, in, in quadri. Quad speak. And you would be talking at whatever speak. And, and.
- [16:14]
B
Models understand your quads. Do they understand their, their.
- [16:17]
A
Well, actually, I should probably say teenage speak because, you know, Teenage speak.
- [16:21]
B
There you go.
- [16:21]
A
You know, I've gotten pretty, it's funny because you don't, you know, like, you lose touch of the teenage speak and then when your kids become teenagers, you then become clued back in on it. But, but, you know, the, the, the language models could translate this for you on, on the fly.
- [16:38]
B
Right.
- [16:38]
A
You know.
- [16:52]
B
In fact, we're starting to see some universal translator products, I think, at, at ces. We're going to see it in droves at ces, which is super cool. In fact, I use generative AI to generate my podcast, not my interview podcast, but I have a newscast that I do as well, and I generate that in eight languages. I cloned my voice and my wife's voice, and we discuss the new, the hot topics of the news in tech. And we speak eight languages according to our podcasts, all completely generated with AI. And it takes me less, less than an hour to, to do all that work. It's truly incredible.
- [17:36]
A
Yeah. And if you're going to want to do that in a real time basis, that's where you won't have time for that to go to the cloud.
- [17:43]
B
So go to a cloud and come back.
- [17:45]
A
Come back. You know, it's kind of why, why there's, I suppose there's GPU and MPUs on Teslas and cars, right? Because there's not the latency. By the time it takes to go to the cloud and then come back, you hit something so you Know, that's why they're, you know, rolling data.
- [18:01]
B
Yeah. All you need is a, a downloading weight symbol on, on your Tesla as it's doing autonomous driving. Just one second. Well, there's a stop sign.
- [18:13]
A
Yeah, hold on. Break. Be like my, I don't want to criticize my mom's driving, but there you go. That's mean.
- [18:22]
B
But the NPUs are really small too. Very low power.
- [18:26]
A
Right.
- [18:27]
B
So I could really see NPUs being put into glasses, watches.
- [18:32]
A
Yes.
- [18:35]
B
This could be really huge.
- [18:38]
A
Yeah, that's. I hadn't even thought about that. But yeah, for sure. Like, you know, as we're trying to, you know, wearing glasses and want to do real time trans, real time diagnosis or recognition of what we're looking at, it'd be great use case for that.
- [18:53]
B
Yeah. Wow. I mean the, the, the, the field is all ready. It's, it's ready for a lot of great new applications out there. So, so tell my audience a little bit more about what you guys do with your company and why we're even talking about AIPC with you guys.
- [19:12]
A
Yeah. So my company makes a product called Systrack. So we work in sort of helping customers understand employee experience of their IT environment, devices and things like that. And one of the big things that people are wondering about is like what applications could take advantage of MPUs, what applications are using MPUs today? So we're one of the intel launch partners on monitoring MPU usage and applications using MPUs to help it understand how to size these things, who it applies to, what workloads could go there, things like that. Really planning how you roll those things out and then once you do, are we using it effectively, are we overloading them and things like that. So really planning system planning around that aspect and being able to monitor the performance of, you know, these desktops and laptops that have these npus on them.
- [20:14]
B
So have you already started gathering data on. I mean, you have deployments out there already and what are the trends that you're starting to see?
- [20:24]
A
Certainly early adoption is a lot to do with video imaging and conference calling and all that sort of stuff that, that's already happening. As I mentioned before, we're starting to see people take advantage of running local language models, local large language models.
- [20:40]
B
Did we just make that up?
- [20:41]
A
Triple. Mmm.
- [20:44]
B
L3M? Is that what it is?
- [20:45]
A
L3M? You know, in. We're going to see more of that. You know, I think enterprises are kind of like trying to figure out how they, how to use the aipc. We get involved a Lot with companies that are trying to figure out like as they, as they replace their desktop fleet, do they, is this the time to buy AI PC? So we work a lot with the OEMs on that, you know, like the Lenovo's and the Dells of the world to help their customers decide like, should I buy a AIPC today and how can I use them? So yeah, I think, you know, it's great for people doing a lot of video conferencing, I'll tell you that. You'll see a dramatic performance increase in team meetings, in Zoom meetings, probably even Riverside meetings and things like that.
- [21:43]
B
When you say performance, are you talking like battery life? Are you talking clarity of video?
- [21:50]
A
All of it? Yeah, battery life, clarity of video, battery life in terms of, you know, network utilization, all those sorts of things. You optimize it.
- [22:05]
B
That's incredible. So this is almost the invisible. People don't think that that's AI.
- [22:14]
A
Right, Right.
- [22:14]
B
Because AI is something I need to talk to. And you know that chat bot, you know, because that's what, that's what ChatGPT brought to the world was the chat bot. Right. The AI enabled chatbot. So this kind of AI is kind of hidden from the end users, but the benefits are huge in, in performance, efficiency and things like that.
- [22:36]
A
Yeah. If you've ever been on a conference call when you've got a bunch of kids screaming in the background and you think that people can hear it, but they can't. Yeah, that's AI in the background filtering that out.
- [22:51]
B
Yeah. I guess most people don't even think about that. That's just that invisible tool. What other kinds of invisible tools are out there that maybe we don't know about, but we're seeing the benefits through like things that we care about, like battery life.
- [23:08]
A
Yeah, I mean, well, you know, if you're using GPU can do the same things, it's just that GPU does them, you know, uses a lot more power to do them. Right. So battery life is definitely an invisible benefit. Yeah. So what we do is we collect data on the, the PC itself. And our architecture is we collect the performance information and we store locally and we process that data on the edge, on the PC itself. And we use the MPU process to analyze and store that information and then that then send to the cloud the analysis versus the raw data, which allows our platform to scale in a cost effective way. Because if you have a solution that collects data and then just sends it in a stream up to the cloud, then you have to use cloud processing to process that data.
- [24:21]
B
So, so I'm also decreasing the amount of data that I'm sending up to the cloud dramatically.
- [24:27]
A
Yeah, you're sending.
- [24:28]
B
So I'm not getting my network, I'm not wasting network bandwidth and things like that. I can start to see a lot of other use cases. I know some other partners are using it for malware detection. Right. They put it directly in the MPU instead of using a GPU to do pattern recognition, to say, hey, this looks like malware. So that's pretty. When you think about it, we can use these NPUs to do a lot of things that before were slowing me down. Right. I hated it when the detection software, detection, malware detection thing kicked in because I could always tell when it kicked in because my machine would go, oh, yeah, yeah. Now it, I don't even notice it because it's happening.
- [25:18]
A
Yeah. I mean, the impact to the security tooling, you're absolutely right. Like the ability to respond quickly, quicker to these, to these threats, you know, rather than having to farm them back to the cloud, analyzed and back. You can do that analysis right there locally. Yeah, that's a great use case.
- [25:41]
B
That's awesome.
- [25:43]
A
And not destroy the performance of the machine, which, you know, traditionally there's this sort of line between like protecting the user from hurting themselves to giving them performance so they can actually get stuff done. Right. So there this line. So if we can make it more effective, you can be secure and be able to do what you want to do.
- [26:07]
B
Yeah, no, exactly. I, I always talked to security people and I said, what's your optimal security footprint? They said, in a concrete bunker with no connection to the Internet, I'm like, okay, it's totally useless, right? Totally useless. With no USB keys to plug anything in. Yeah, that's the ideal.
- [26:31]
A
With a 3270 VT100 terminal. Right, exactly.
- [26:37]
B
That's exactly right. But you know, that's not reality. So I really love that we've got another processing unit that can be tailored to specific use cases like npu. Do you think we're going to see other types of processing units being developed for specific types of use cases? I mean, MPU is still kind of broad, but do you see other ones being developed?
- [27:04]
A
Well, you started to talk about glasses and some other IoT devices. I mean, I could definitely see. So a PC is a general purpose computing device. So I think you probably need a general purpose MPU device that's optimized, that people leverage. But I think when you start talking about IoT or OT, you know, operational technology, people using, you know, video cameras and all sorts of things. I could definitely see specific workloads optimized on those devices or single purpose devices versus general purpose devices.
- [27:38]
B
Like, like for a camera, object detection.
- [27:41]
A
Right.
- [27:41]
B
Or threat detection. Right. You know, we, and there are specialized processing units that do vision.
- [27:50]
A
Right.
- [27:51]
B
That's, that's what a lot of the autonomous vehicles are starting to use, is specialized processing use. So cool. This is really cool stuff.
- [28:00]
A
Yeah, we're sort of scratching the surface, I think, on where all this stuff goes.
- [28:06]
B
Yeah. AI is not bringing computing to an end, it's unleashing even more, it sounds like.
- [28:14]
A
Yeah, I mean, nobody knows where it goes in the end, but certainly we've seen these cycles in technology. Right. The invention of the Internet, you know, that didn't bring businesses to a close. I mean, it really optimized things and brought up new ways of doing things. I'm sure the same will be with AI, you know, in software. I mean, that's, it's affecting software a lot. But, you know, we, I think I've read something where there was millions and millions of hours of backlog software projects that never got to people. Never got to. Now people can get to more stuff, automate more things, you know, so it's, it's going to be real interesting where it all goes. Look forward to what the kids are going to be doing, you know, like in the future.
- [29:04]
B
Oh, speaking of kids, what? Your kids started college this year, right?
- [29:08]
A
Yep, yep.
- [29:10]
B
All right, so what are they studying? Are they all studying all different subjects?
- [29:14]
A
Yeah, I mean we've got an one studying sort of economics, math, a chemistry, a biology and a biomedical engineer. Are the four all sort of sciences? Nobody doing computer engineering, which is what I did.
- [29:34]
B
They'll come around to it. Give them some time, Dan. They'll come back.
- [29:37]
A
And my wife's a lawyer, so we'll see where that goes, you know, so she's pushing them hard that direction.
- [29:44]
B
Yeah. Maybe you'll get some patent attorneys in there, you never know. So, hey, Dan, this has been great. If people want to learn more about you or your company, where do they reach out?
- [29:55]
A
Just lakesidesoftware.com, you know, contact us there.
- [30:00]
B
All right, perfect. And you're out on LinkedIn as well, I'm assuming, Dan?
- [30:04]
A
Yeah, yeah.
- [30:06]
B
All right, perfect. Well, hey, Dan, thanks for coming on the show. This has been a great conversation.
- [30:10]
A
Yeah, thanks, Darren. Thanks for having me. Congratulations on the 10 Kids. That's, that's amazing.
- [30:15]
B
Hey, thanks. And, and for Quadruplas, man, I, I, I bow to your wisdom on that one. That that's tough.
- [30:24]
A
Yeah, well, it's fun.
- [30:29]
B
Thanks for listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite, favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community@patreon.com embracingdigital where we share bonus content and you can always connect with other change makers like yourself. You can always find more resources@embracingdigital.org until next time, keep Embracing the Digital Transformation.