← Embracing Digital Transformation
Loading summary
Transcript84 lines
- [00:00]
A
One macro change that you probably familiar with is the transition from VMware to a more like open source bare metal or kubernetes based containerized environment. Kubernetes. So this is one trend and the other trend is AI. The two combine together. Right. So the kubernetes kind of become the default, let's say orchestration platform and the workload become AI workload.
- [00:24]
B
But welcome to Embracing Digital Transformation, where we explore how people process, policy and technology drive effective change. This is Dr. Darren, Chief Enterprise architect, educator, author and most importantly, your host. On this episode. AI Data and the Future of Infrastructure with Aran Kerzner, CEO and founder of Lightbit Labs.
- [00:58]
A
Iran.
- [00:59]
B
Welcome to the show.
- [01:00]
A
Yeah, thank you, Darren, for having me. Yeah, it's great to be here.
- [01:04]
B
Hey, I'm really excited about the topic today because it's very practical and it's kind of the foundation that makes AI move forward, which is the data. I need data to make all this happen. But before we dive into data and AI, everyone that listens to my show knows that I only have superheroes on the show. And every superhero has a background story or an origin story. So Hiran, what's your origin story?
- [01:34]
A
Okay. How far you want to go?
- [01:37]
B
As far back as you want to go. If it's interesting, that'll be great. If it's boring, I'll cut you off and we'll move forward. Okay. Okay, okay.
- [01:43]
A
No, just joking. So yeah, Eran Kirzner, I'm the kind of founder and CEO for Lightbeats and I started my career more than 20 years ago. You'll be surprised. Actually, I was started being doing a silicon chip. I was an architect in Motorola Semiconductor. You remember back then there was kind of two giant. There was intel and Motorola.
- [02:08]
B
Motorola, that's right.
- [02:09]
A
Yeah. These days people do not appreciate. Do not even remember that. Motorola Semiconductor together with Power PC Somerset, if you remember, they were kind of the king. So I was a CPU architect.
- [02:22]
B
Did you work on the 68,000 chip?
- [02:24]
A
I work on the Powerquake and Powerquick 2. This kind of. Yeah, that was such a.
- [02:30]
B
Those were great chips. I can't say they were better than intel chips because I work for intel, but they were better than intel chips.
- [02:38]
A
I can tell you, you know, when I just graduate, I had to offer in my hand intel and Motorola. And as a kind of graduate from school, you see which one I can go right. Both of them are great. Both of them building the best view. And it was a really tough decision. And you know, it's like the, the movie of kind of swinging doors, things like that. I don't know how my life would be if I was intel or in Motorola, but I choose, I choose. I choose Motorola. There, you know, it was kind of a combination of going to compute, networking and storage because it was kind of. And those, those CPU basically if you remember they power even the Apple. We just talked about Apple before. So they power the Apple machine and then Apple flipped to you know to intel and then they flip their own silicon.
- [03:29]
B
They flip to their own.
- [03:30]
A
Yeah, this was an exciting time in the kind of the. In the silicon building kind of CPU and I kind of learned everything that you need to learn about compute, storage, networking because it was all together. Later on I kind of. I moved to a startup called wintegra. It was people that actually spin out you may say from a Motorola semiconductor. And not unexpectedly we have a site in Austin. So I spent fewer in Austin, Texas and we had a site in Israel and we built one of the best kind of network processors. Remember there was a time there was a hype about network processor I remember and it was very successful and eventually with the startup was acquired by PMC Sierra. But before that we basically kind of have a very kind of dominant position with the customer like Cisco, Ericsson, you know, Nokia, Siemens and you know kind of SoftBank and others. So it was pretty exciting time of being in the leading technology and it was leading because this actually what was the foundation for LTE for everything that every base station, ASN Gateway, all of that. So I got kind of the opportunity to be in kind of the compute intense, you know how to kind of process and kind of CPO intensive and then moving to networking and everything related to communication. And then as part of PNC Sierra the CEO asked me kind of to be part of a new group that incubation team and leading kind of some of the development of the Flash controller. So shift from compute to networking to storage. So we basically built the first NVME controller that went to the market. And this actually what led me later on to starting Lightbit. So building the first NVME controller went to the market. It's used by the top hyperscaler by most of the storage vendors and it. It was a phenomenal from that perspective really being in the. So I had the opportunity or the kind of this why I was joking of. I don't know how my life would be if I would start here or here. But I had the opportunity to be in the bleeding edge technology in each one of these domain and Now I feel the same thing with everything related to AI and kind of data centers and NEO cloud, but I guess that's the next part of our discussion.
- [06:04]
B
Well, well, this is really interesting because you're, you're actually very un. And your superpower is that you've got to be. Because most silicon engineers get stuck into CPU or into network controllers or into. You've had the opportunity to handle the whole compute domain, which is compute storage network.
- [06:27]
A
Exactly.
- [06:28]
B
Which is incredible. And now you're actually delivering products to end customers instead of just focusing on a piece of silicon, which you and I both know. When you're down in the architecture of a piece of silicon, you never talk to customers. Never. Because customers are so far removed from what you're doing it because you gotta focus. So this is one incredible background. I love it. So let's talk, let's, let's dive into AI. AI has disrupted everything.
- [07:01]
A
Yeah.
- [07:02]
B
Even though it's been around a long time, it's really only been the last three years since Chad GPT launched November 30, 2022. That's what really brought all the attention to the world. And now we're seeing an explosion of AI. What effects has that had on your startup here?
- [07:36]
A
Yeah, so it, it, they have multiple kind of impact of the startup of course, you know, from the kind of internal activity, external activity and customer engagement. Right. So if you look at about light beats, we have three type of customers. Okay. By the way, I haven't mentioned it but we, you know, we at Lightbeats we invented a new protocol. Okay. It's pretty unique for a startup to invent a protocol. It's called nbtcp. Yeah. And we haven't done that by ourselves. We actually had a few strategic investors like intel, like Cisco, Micron, Dell and others that help us kind of to bring it to the market. Right. So you ask how we did so it kind of impacting us. So from multiple direction I will tell you. So we have three type of customers. One set of customers. E commerce, very loud E commerce. And over there it's all about fraud detection, analytic AI processing, you know, and some of our customer, you know, without naming exactly. Kind of customer name have like 500 million customers. You can just imagine the intensity in their data center. Okay. Think about, think about if you go to kind of the, you know, Christmas time like we have now, or think about kind of you going to kind of Black Friday, the amount of intensity on the data center. It's all about transaction. I don't want to be an electron running there in the data center. It's kind of, it's huge. So this is one type of customer. The second type of customer we have is financial, okay. And it's including top bank, you know, insurance company, hedge fund, fintech over there. It's really about kind of the tight kind of processing, low latency, mission critical type of application. And then the last segment, which probably will be the topic most aligned to our discussion is cloud provider and new cloud providers. Okay, so those cloud providers everybody's looking to now to kind of to host AI workload, right? If in the beginning, you know, like just go like five years back, you know, you had very kind of multiple general purpose public cloud, right? Of course we have the large public cloud. Everybody familiar with Amazon.
- [10:02]
B
Right, Right, right, right, right.
- [10:03]
A
Microsoft. Now there is kind of a new domain, a new segment of tier 2 cloud provider and some of them being called Neo Cloud which focus on, on GPU base, focus on AI. And this is where we are playing. So how it impact us. First of all, a lot of our development today is done with, with AI tools, okay. It meaning things that related to code development, code review, different security analysis. So there's a lot of AI impact kind of in the day to day that increase the kind of utilization and increase the productivity and you know, adding kind of additional kind of consultant to every engineer. From our customer point of view, you see a change in the workload, okay, the change in the workload is coming in two form one. There is a one macro change that you probably familiar with is the transition from VMware to a more like open source bare metal or Kubernetes based containerized environment Kubernetes. So this is one trend and the other trend is AI. The two combine together, right. So the Kubernetes kind of become the default, let's say orchestration platform and the workload become a workload. But yeah, let's take it to the next level.
- [11:22]
B
So let's talk about the workloads a little bit because those three industries that you talked about are data intense, but.
- [11:30]
A
They'Re not.
- [11:33]
B
It'S small amounts of data, it's not big blocks of data. Right. Even the AI training that you're doing, you're talking about small bits of data that are, that are being streamed through. How has that changed? Has that changed the, the profile of the data moving through the system? Where in the past maybe I loaded a whole database up, right? Oh, big movement of data. Now it seems to me like I'm moving a lot of small packets Are you seeing that? Or, or do we still have the, the same mix, the profile mix that we had before?
- [12:16]
A
You know where I'm going with that?
- [12:18]
B
Iran.
- [12:19]
A
So it's kind of. So there, there's kind of. If we double click on AI specifically. Right. So you need to kind of. There is two major activity. All of us familiar with training, and there is inference.
- [12:31]
B
Okay, right, yeah.
- [12:33]
A
Training and inference is different by behavior. Right. Training, you have access to all the data ahead of time. Say, okay, I'm going out to train of all the video, all the newspaper, all the kind of insurance kind of contractors that we had. If you are, if you are bank. So you have all the transaction, you have all the data ahead of time. And it's a game of how fast you bring the data in. Okay. And it take you time. The training process may take like weeks or months. And it's all about throughput and readability. Super important. You don't want your kind of job to stop and then recover everything. From the beginning, you would be able to checkpoint and recover. Okay, so it's about throughput and large chunk in this case. Okay, the chunk is that you're bringing slightly.
- [13:23]
B
The chunks are big for training.
- [13:24]
A
You bring it into. Close to the gpu. You bring it. The CPU is the ones that initiate it. You get it into the DDR close to the GPU. Then the GPU get it even closer to the HBM. All of us familiar with kind of DDR, there's no DDR. Okay, let's wait 24 weeks. Exactly one year. Okay, I can get you DDR five. So you know that. So this is one aspect. The second aspect is inference. Inference is totally different. Inference, you have the wait, wait, wait.
- [13:54]
B
Before you go to inference. On the training side, I'm reading in big chunks, but when I do a checkpoint, is that a big chunk out or am I only sending small pieces back out? Because that makes a huge difference in the way that I architect my storage.
- [14:13]
A
Correct. So it will still be kind of a. It will not be petabyte or megabyte, but it will be a chunk of like 128k or so on. Yes.
- [14:23]
B
They're still pretty sizable chunks.
- [14:25]
A
Yeah, yeah. You will, you will send it kind of in a, in a, in a chunk, and then you have to register it from all the GPU together. You make sure that all GPU in your cluster actually register, and then you can continue to the next checkpoint. Okay. And you can have a checkpoint every 30 minutes, every one hour, depending how kind of power need you Are or kind of how critical your job. And it's a balance between stopping a job and, and continue kind of you need to do the, the basic trade off. But.
- [14:53]
B
Okay, gotcha. So that's training.
- [14:55]
A
Yeah, but, but you need to remember that for every AI activity there is additional accessories, I would say so there are databases that running in parallel that allow you to adjust the parameter to allow you kind of get some kind of a different tuning. And this will be let's say regular database and that could be a small chunk and it will be intensive read, mainly intensive read, A lot of intensive.
- [15:20]
B
Read, not as much write.
- [15:22]
A
Correct. So this will be kind of the macro processing. In training, When you go to inference, it's different. Right. So you have the model. Okay. And now it's very interactive. Okay. Think about an environment that you are hosting millions of customers now. Okay. And you know, it's kind of, let's say you're running your, your chat GPT or in your bank you're running some kind of an agent. Tons of agents. Yeah. That's running a chatbot. All of them are running in parallel. So what do we see? It's very interactive. It can be small transaction because it's interactive. Now latency is the king. Before, you know, before that it was like throughput was the king. Right. How fast you can get me all this chunk in so I can crunch it in my gpu. Now it's not just about GPU crunching, it's about kind of the latency, the interactive session. I have one agent put a task, another agent waited to the result. Right. And now think about thousand like them or more than that. So you need to manage multi tenant environment, multi context environment.
- [16:42]
B
Okay.
- [16:43]
A
So it's not one stream of data, it's very different. It's a lot of stream of data. It's kind of multiple contexts, sometimes multimodal in parallel. And the latency have become more important. And it's also about now how you can get your GPU very efficient by just doing the job you need to do with waiting to the data, waiting for the storage.
- [17:07]
B
Okay, well, so here's a good question I have on this. GPUs are not really made for that kind of interaction. Right. They're mostly made for big chunks of data. Right. Coming through. Not a lot of little small ones. So why even use a GPU in this case? Why not just use a CPU to do your inference?
- [17:29]
A
Correct. Correct. You're absolutely right. You will see some of the. So I would say even absolutely correct, what we see now in the transition period is that people using the training blueprint, okay, and use it for inference. But as inference becoming look, people will look to make inference more cost effective. You know, the cost per token, the power per token, all of that will come to play. People will start to see, okay, how I can customize my inference environment, how I can make it more efficient. And it's more than that even because when you talked about training, you talk about very large environment, maybe a set of customers that will be expert in training. But inference will happen everywhere. Everywhere. Right. So there's no one size fit all. Think about inference in the edge. Think about inference in your office. Think about inference on your mobile device.
- [18:28]
B
Well, on my aipc, right, I've got on my aipc.
- [18:32]
A
Yeah, yeah, exactly. And you just want to do inference for your blog or your. You will do interface for a data kind of with a different set, smaller set. So I don't think there will be kind of one solution fit all. And we are going to see kind of fast forward maybe six to 12 months from now, custom silicon custom devices. You will see workload offload to the CPU which is much, much less expensive, less power consumption. So it will be a combination of gpu, CPU and custom devices that will host all of that together. Right. And when you have your orchestration environment into the kind of the inference, the orchestration environment based on the input will have to select the right inferen.
- [19:20]
B
The right environment. Right. Whether it's an inference or training or Correct, Correct.
- [19:23]
A
If you inference now video or text or audio, it will get different treating okay, the type of the models that you want to get.
- [19:34]
B
This is really interesting because it kind of flies in the face of the whole concept of cloud computing which was I'll just throw a generic set of hardware in there that all the workloads can take advantage of. Right. But what you're talking about here is specialized type of setups infrastructure, whether it's a CPU plus GPU for training or cpu maybe an NPU for inference and video inference. Maybe I have a vpu, a visual processing unit in there. So do you see that we'll move into more heterogeneous type of configurations in these cloud service providers or even in my own data center. Right.
- [20:19]
A
Yeah, I absolutely do. I think we are kind of two step. Okay, step number one people just let's get it work, right? And they will take the blueprint and many times the Nvidia blueprint and we use it for inference. Now when they start to kind of to make and if you look at the all the Neo cloud they're also looking on kind of to be to go one level up in the food chain. Right. They want to become inference as a service. Okay. Instead of selling equipment and letting you.
- [20:48]
B
Host, they just want to be an inference service.
- [20:50]
A
Yeah, they want to be an inference service. You just come, I will give you an API. You have a route open route or whatever and it's about kind of token per second your API become now token API. Okay, yeah.
- [21:06]
B
Which is an easy API.
- [21:07]
A
Yeah, yeah. Metaprompt in, metaprompt out. And now if I'm a cloud provider and it's not just about selling the equipment and scale get access to GPU making it cost effective. Because now my margin will be based on the kind of how many tokens per second I could create, how much power consumption I need to get for generating you the thousand token and so on. So when those cloud providers and it's happening now those cloud provider basically we start to look at the economic of this. They will say maybe I can do this work with which is cpu. Maybe as you mentioned, I can be VPU or NPU and then I'll have different kind of environment. And because I'm on the entire inference whenever I get an input I know exactly which data set, which element I want to use. Right.
- [22:02]
B
No, that's pretty cool.
- [22:15]
A
This is why we at Light Beats are pitching about kind of the everything needs to be software defined. Okay. And I explain you why fully agree because if you go with kind of the very kind of traditional model, okay. I put an appliance that will satisfy all my need. That means that you don't have the agility in the data center. Right? You mentioned it before. I wanted to build a data center that I have kind of all the flexibility to shift workload between the compute into storage into gpu. So I need kind of an environment that is very elastic.
- [22:48]
B
Okay, Right.
- [22:49]
A
It can be elastic at the time of the day, it can be elastic kind of the time of the year. And of course I can build everything to the worst case. But all of them know it's not economic. You know, building a data center today become more expensive than building a fab. Right. I lost crazy in the past to say okay, building a fab. Do you have a 7 billion dollar building a data center today? Like 200 megawatt, 300 megawatt, same number, 7 billion dollar. Okay. You need the energy, you need to put kind of the all kind of systems. I mean yeah, environment, all of that. So you need to be Kind of more sophisticated and more kind of, I would say control on the way you're building it. Otherwise it will, you know, if you're building it in the wrong way, you build a cloud, okay, you get state of the art GPU, guess what, 18 months later, this obsolete already.
- [23:44]
B
Yeah, you got thrown away.
- [23:46]
A
So. Right. So there's new generation. So you need to build your environment in a way that it's getting more like kind of sophisticated agile elastic cloud. And then to find the best way to host your environment. And it will definitely, as you mentioned, be a trigonius. It will be a combination of compute and storage and GPU and networking of course. And all of that in my vision should be a software defined.
- [24:12]
B
Okay, so I have a question on that. I'm a true believer of software defined infrastructure. It's part of the digital transformation architecture I've been working on is software defined infrastructure for storage. We've talked about compute being specialized, right. For the different types of workloads. What about storage do we see? Because right now, I mean we've got two major flavors, file based and block based. Right. Or object based. Right. Object based and block based. Do you see a change in that as well as we get into different workloads that we've never seen before, especially at high volume that we're seeing like with, with inference and the high volume in training. I mean training consumes a lot of data. So are you seeing a change in data architectures or storage architectures because of this as well, or can we lean on some of the things that we've been using in the past?
- [25:13]
A
So it's an excellent question and I think people are sitting today in multiple startup, including at lightweight and trying to crunch this question. Okay. And to see kind of how kind of data center from the different kind of training model or inference model will look from the stored point of view in the next like 12 to 18 months. And let me double click on that. Okay.
- [25:41]
B
Okay.
- [25:41]
A
So when we built kind of the first kind of NVME controller and we kind of build it kind of, it was kind of the evolution or revolution, you may say, from going from spindle into kind of flash. Right. And it was very clear what we're trying to solve. Right. With Spindle, it was all about sequential, you need to have kind of a spin and you know, it was, you know, best tailored for the single thread and then you know, as kind of database and digital transformation you mentioned. And all of that came in a containerized environment. It was clear that it's all about Random data, small chunk. Right. And you need something different. And this was kind of leading to building the, you know, nvme, NVME over fabric and building kind of the new generation of Flash controller that was fit for kind of latency, random access versus sequential and kind of specific durability and endurance time and so on. Now when you look at the GPU era, it's basically, it's a great question because today the GPU get all the data from the cpu. Okay. The CPU is sitting as a broker, right?
- [26:55]
B
Yeah, yeah.
- [26:56]
A
He's trying to predict what the GPU will need. He's pre fetching it into the GPU and then the GPU crunch it. Okay. So it's kind of there is the duality between the CPU and the gpu. Okay. And if you think about kind of take it to the next step and thinking about customized kind of xpu, let's call him not just GPU for different infrastructure. Yeah, yeah, yeah, yeah. Maybe we need a direct interface between the gpu, the XPU and the storage and the memory. Okay. Maybe there is a place for that. And I think the answer is yes. I think we need to come and start to build kind of a new data structure, a new kind of interface that will make it much easier to the gpu. You mentioned in the beginning of our talk, you say GPU was not kind of adjust or plan for this type of workload. Right. And this is true. GPU cannot run a very sophisticated file system. GPU cannot run a sophisticated kind of distributed object store.
- [28:04]
B
So it's not made for that. Right. It's made for massively parallel computing.
- [28:08]
A
Exactly. And you can say, okay, I can put some kind of sophisticated like memory controller. Remember in the past you had like just a memory controller. So we can have some sophisticated storage controllers in front of the gpu. So I think it will be the two things together. A maybe some of the work can offload into SmartNIC environment. All kind of. And some of the work need to go back to the XPU or GPU and it will lead eventually to some different data structure. I believe not necessarily for training, but more for inference. That's my business. Yes.
- [29:00]
B
So is that what Light Light Bits is working on is a new protocol for this? Where do you guys fit in this? In this?
- [29:09]
A
Yeah, so we definitely kind of, we definitely crunching this problem. Okay. And you know, we already provide kind of solution today to, you know, as I mentioned, the cloud provider and NEO cloud to kind of run the old workload, the existing workload today. And when you look at inference, it's rag and it's vector DB and it's the inference process. All of that together, part of the package. And there is video inferencing and there is kind of text inferencing. Each one of them is different. And the next step is to provide with an optimal solution for inference and in a more compact environment. And yes, we definitely working on something like that. When you invite me to a, to another podcast maybe. Yeah, six months from now we can go deeper and I can.
- [30:01]
B
All right, that sounds, that sounds good. We'll have you back in six months. So do you see things getting better then as far as OPEX cost? Because like you said, the CPU is the quarterback, right? The CPU is the guy in there directing traffic and all that. And if I can pull that out of the CPU and put it in possibly an accelerator chip or memory controller or something that or storage controller that talks directly to the gpu, that will save an inordinate amount of time on moving data around. Correct. Because I don't have to go through a CPU and get context switched out and all the things that come with that.
- [30:50]
A
So basically what we are doing now is helping people to redesign their data center of the future. Right. Because as we mentioned, what they're using today, what available off the shelf component. Right. And in a way you would like to take the CPU out of the data pass today the CPU is part of the data pass. While the CPU should kind of mainly focus on the orchestration, the control pass, on the kind of more sophisticated kind of software kind of aspect to execution which the GPU is very routine on the type of execution. And I think this will happen and in lightweight we definitely going to be there to offer solutions that help this to make it to become affordable.
- [31:35]
B
I would say that's awesome. I mean we've done this before with network. With network, right?
- [31:41]
A
Yes.
- [31:41]
B
I can go network to storage directly. I don't even have to touch the CPU right. Through accelerators.
- [31:47]
A
Right. You have rdma, you have kind of. Yeah. So yeah, there's no reason you will not be able to do it here as well.
- [31:53]
B
This is, this is exciting. This is exciting stuff. There's so much happening in this space. If people want to find out more Iran from you, where do they go to find out more?
- [32:07]
A
Yeah, so very easily. Right. They can even contact me on LinkedIn. They can contact our website, lightbeatslabs.com and we more than happy to educate them and help them to design their system. We basically always feel that we are A partner of our customers. We're working with them together on the requirement we understand exactly what they need and we have the luxury of understanding, understanding. You know, that's part of the DNA of the company. Compute, storage and networking. Oh yeah, you do? Yeah. We are not just providing you kind of a point solution and tell you, okay, go and solve all the problem. We are more than happy to talk with you about the entire solution. Okay. How to architect your network, how to architect your storage, how to kind of optimize your compute. All of that together and helping our customer to get the best kind of state of the art solution. We understand that we are in transition point and this is why software play kind of large part of that because you need to adjust, you need to kind of do you know your software as it is today will be different maybe six months from now.
- [33:10]
B
Absolutely. Especially as agentic flows start coming up. You're going to see different data patterns and all that going on. Iran, thanks for coming on the show. And for my listeners, you got to tap into the superpowers. These are incredible superpowers. There's not a lot of people that can, that can go at the silicon level, that can go from CPU or compute all the way through network and storage. There are not a lot of people in the world. So reach out to Aaron and Lightbit Labs. Thanks for coming on the show. I appreciate it.
- [33:43]
A
Yeah, thank you for having me. It was great talking with you, Darren.
- [33:52]
B
Thanks for listening. Listening to Embracing Digital Transformation. If you enjoyed today's conversation, give us five stars on your favorite podcasting app or on YouTube. It really helps others discover the show. If you want to go deeper, join our exclusive community@patreon.com embracingdigital where we share bonus content. And you can always connect with other change makers like yourself. You can always find more resources@embracingdigital.org until next time, keep embracing the digital transformation.