← Reshaping Workflows with Dell Pro Precision and NVIDIA RTX PRO GPUs
Reshaping Workflows with Dell Pro Precision and NVIDIA RTX PRO GPUs
VLLM, Inference, and the Next Era of Intelligent Workflows
00:26:52
Get an inside look at VLLM, the open-source engine making LLM inference faster, scalable, and more efficient for local and cloud AI deployments.
Loading summary
Transcript55 lines
- [00:00]
A
Foreign.
- [00:05]
B
Welcome to Reshaping Workflows with Dell Pro
- [00:08]
A
Precision and Nvidia, where innovation meets real world impact in high performance computing. Welcome back to another episode of Reshaping Workflows with Dell Pro Precision and Nvidia RTX Pro Pro GPUs. I'm your host, Logan Lawler. Today, very excited episode that we're going to have. There's a term that gets knocked around a little bit in the AI world called vllm. And I'll be honest, like I was aware of it, I kind of understood it, but the meaning and what it actually does, you're gonna get to hear kind of firsthand from my guest today, kind of the ins and outs of vlm, what it does, why it's important, how it can help you scale your AI activities. So with that, I have Kai Chau with me. Kai Chou, welcome to the show. Take a second and introduce yourself a little bit on your background, what you do for VLLM and anything else you think might be relevant.
- [01:05]
B
Thanks for the introduction, Logan, and glad to be here. So I'm Kai Chao. I'm working on the open source project called vlm. So it's basically an open source software allow people to run large language models everywhere with any sizes. If you think LLM as kind of like an intelligence, so people train it like. Yeah, like big companies, they will train these powerful models. But then after training, people would deploy the models somewhere to run it massively for all the people. So for example, when you are chatting with a chatbot, when you are writing code with cursor or crowd code, whatever just called inference. And inference is increasingly difficult because of the scale, because when, after the LLM becomes very popular, people all want to run the LLM to change their lives, right? So that's how AI is shaping our lives. And VLM is one of the underlying power to accelerate these LLMs and make our life better.
- [02:18]
A
I love that. Great explanation. So vlm, I'm assuming, does the V stand for virtual?
- [02:26]
B
Yeah, the V stands for virtual. Because the original idea of VLM is basically we do complicated virtual memory management similar to the previous operating system. So it's like managing the large language model, like a process in the operating system using the virtual memory to allocate the memory for kvcache.
- [02:50]
A
I think everyone on the episode's very familiar with like, you know, inference. So you know, VLM is specifically related to the inference after you've kind of deployed a model. But my question is, is that with virtual, when you have the V, Right, we Have the V in front of the LLM is the LLM, let's say I downloaded, you know, ChatGPT OSS 20 billion, right? Is that actually running installed locally on my system or not? When you use vlm, when you use
- [03:18]
B
willm, you can, I mean from end user perspective, you can download the model locally and then you can start VLM and it can serve the model, right? That's one of the approach, you can use willm. There's another approach that some big companies, they can use willm to serve models in the cloud and then embed that service into our everyday life. So people might not even be aware that they are using wlm. So we have many big users. So for example Amazon, if you open your phone and then open the Amazon app, when you browser something, you want to buy something, they have a bot called Ruffus that's actually also running on Will amp.
- [04:05]
A
So let me kind of step, let's start, let's split off the two kind of the cloud versus the local, you know, conversation. Right? So, you know, kind of getting into it, right? Like, I mean model sizes, I mean, are kind of shrinking a bit, but they're also growing, right? Like I think the. Was it Kimmy 2 or whatever it is, it's like massive, right? It's just a massive model depending on how it's quantitized. Like it's 200 billion plus, right?
- [04:29]
B
Yeah, 1 trillion parameters, it's massive.
- [04:32]
A
So would VLM in a situation where if maybe I only have 96 gigs of vram on my PC, would VLM allow me to run a model that far out exceeds the size of the VRAM that I have and still be able to run that model?
- [04:48]
B
Usually we require that you have a powerful GPU to at least host the model. So yeah, if you want to run a 20 billion parameter model, just like GPT OSS, then depends on the quantization
- [05:04]
A
format you use, like FB4, FB16. Well, I'm kind of saying, because I think that's where a lot of people get a little confused with VLM in general, right? Is that it makes it sound like it is virtual, which it is in a sense. Right? But it makes it sound like, hey, I can take a model that is too big to run locally in the VRAM of my system and then be able to still run that locally through vlm. And it does some magic to make it work. But that's not the case.
- [05:34]
B
Correct, that's not the case. So this term virtual does not mean about the model weights. So when you run the model, there are actually kind of like two parts. One is about the model weights. It is like, yeah, you can tell from the model. So for example, GPT OSS 20B is that it has 20B Boolean parameters. So this weight is static and there's another part that's growing when you run the model. So for example, with the same model, you can ask him different questions, you can give him different contexts. So when you are writing code or when you are like doing some daily life, he, you have different context. Right? So this context, after being processed by the model will become the internal states called kvcache. So because how the model runs nowadays, these models autoregressive, meaning that they will turn the input into some internal state and then keep updating and inserting the new states. So these two portions like model weights and KVCache, they can be kind of like comparable or the kvcache can even be larger as people entering the agentic world that they are feeding more and more context to the model. So the term we actually means how we manage the kvcache dynamically. So we use some kind of like virtualization technique, similar to the operating system that we virtualize the rest of GPU memory and manage the KV cache there. So which means you need to have a GPU larger enough to hold the model weights and then the rest part of the memory are managed by wheel and to virtualize to utilize it for further context.
- [07:24]
A
Let me break it down. I am obviously not clearly as smart as you because it's, you know, I'm enough, but probably not in this conversation. What I'm hearing is the real value of VLM and is if you were just to load that model straight in, you know, whatever the size is, it is what it is. Soundsworth VLM is kind of with the KVCache and kind of virtualizing the RAM a little bit, you can actually squeeze a little bit more into a GPU than you would have been before, is what I'm hearing.
- [07:53]
B
Basically, yes. And especially when it comes to the cloud serving case. Well, one instance need to serve multiple users. So these multiple users, their requests are batched together. So we need to have an efficient way to manage the kvcache across different requests. That's where previous systems fail to resolve.
- [08:14]
A
Okay, so it really sounds that the use case, I mean it can run locally. Right. But the real value you're seeing is from, you know, a cloud or like a data center type deployment is where it's, you're serving a singular model kind of one to many is what it sounds like.
- [08:31]
B
There are kind of like two extreme use cases. One is what we call throughput that you care about how many tokens you generate, you don't care about the latency, you don't care about how fast you get the outputs. That's usually when people are doing training, they are generating the output, maybe they are doing some distillation or reinforcement learning training. Well, the total amount matters. And there's also like online serving case where people care a lot about the service level objectives like how, how quickly you get the request and how quickly you generate each request. Yeah, so there are different cases which are both supported in Willem, but in general the serving case is more latency oriented. When it comes to the local side it also matters more like in terms of latency. But I would not say that local is only for batch size one, the only one request is running recently there are very popular like the cloudbot.
- [09:34]
A
Oh, I've heard all about the cloudbot. Who has it.
- [09:38]
B
Yeah, you must know this, right? So you have a local agent, local app running in your laptop and when you ask is something, it can actually like spawn multiple agents. Right. And these agents can, can all be hosted by Will. So even if it's a local use case, we only have one end user. We can have multiple intermediate requests running. So that's actually you also need to have some batching internally.
- [10:07]
A
Okay. So it's not necessarily the users, it's the kind of concurrent processes or requests is what it really sounds like. Okay, you know, that makes total sense now. And I don't know, you know, how much you've been involved with this, but you know, Dell, you know, for a long time, you know, I work kind of in the workstation market, right. So think precision workstations, sets on your desk, mobiles, high GPU compute. But recently we launched, you know, on the backs of Nvidia, the dgx. They launched the DGX Spark and we launched the Dell Pro Max with GB10. So I know I sent you out a couple units that are going to kind of the VLM and like to test and stuff like that. So on a use case like that, let's kind of talk through it. So like the use case agent box makes kind of total sense, right? Or you rock Clawbot. You're like just letting it go to town, you know, organizing your email, everything else. But it's something like that with 128 gigs of, you know, integrated RAM. Something you could see as maybe, maybe, maybe not. It's not a Server, obviously, but something that could handle, like from a VLM standpoint, how many concurrent requests could something like that hold? Like, I mean, I know it's hard because you don't know what the request is, but let's just say it's running like a PLA bot or like, let's say a bunch of just, you know, inference requests or, you know, chat requests on a certain model. Right. Like, what could VLM enable that box to hold in terms of concurrent requests?
- [11:33]
B
I think to get the number in general, we can think of like, how much VRAN you have, that's the total GPU capacity. And then we can look at how large is the model. Like you can check it from hugging face page, like how much is the storage? And then you can know the rest of the kvcache you have. And then every model you can calculate from the model architecture how much size would the KVCACHE be for one token? And then after the division you can get like, for example, can you run the model with 100,000 tokens or you can run it with 1 million tokens. So that's roughly the number. And then if people use all these like LLMs, there are this concept of tokens, right? All the inputs are tokenized into tokens. So you can measure how long you can use the agent by checking the number of tokens.
- [12:31]
A
So there is a kind of a formulaic approach to it.
- [12:34]
B
Yeah, sure. And if you use Will we have some. If you are an end user of Willem, we have lots of logins and you can see in the login that you will see that how many tokens you can use.
- [12:47]
A
Interesting. All right, well, let's kind of go to the next, like, let's say we've deployed, you know, vlm and you might not have this data. If you don't, totally fine. But how have you seen, you know, VLM perform from like a benchmarking standpoint across like different context links? Right. Because, you know, some requests could be very short, very quick, low amount of tokens. But something could be much more complex. Right. Does VLM perform the same across, you know, a large context length request versus kind of a lower length? I'm just curious if it, if size matters at all with vlm.
- [13:20]
B
I would say VLM is designed for running the large language models in general. But, but I think for different kinds of workloads, people care about different characteristics. So usually there are many flags in Willem that you need to tune. Say, for example, if you are running a very short request, I think people do it for for example recommendation systems like yeah. If given the resume and then select some of the candidates. So this is. These are kind of like the pre fill heavy and then for agentic tasks that people want to do tool calling want to do syncing especially thinking right. I think Last year the ChatGPT01 release and then deep seq R1 release people are starting to see the model need to sync for a long time before answering you the questions. So these are kind of like we call decode heavy workloads. So we have different feature sets for these scenarios and you need to be an expert to turn on these knobs.
- [14:24]
A
Okay. So it's not something that myself. It's like your level of skill to be able to do kind of this. This level of like tinkering it sounds like. Or this adjusting.
- [14:36]
B
Yeah. If you want to tune for your workload usually you need to be some experts to learn the meaning of the knobs. So it's like operating a very complicated machine.
- [14:47]
A
Now I know that VLM and you look at your. You put it on kind of the Nvidia like Grace Blackwell. I'm sure it runs on Nvidia H1 hundreds, two hundreds. Does that also apply across other Blackwell GPUs like the Blackwell series, like the 6000, you know, Blackwell RTX Pro and down the stack. Does VLM currently work on that as well?
- [15:08]
B
Yeah, sure. VLM works for all the black wheel, not only the data center one, but also the desktop one and also the GB10, the Grace Blackwell.
- [15:20]
A
Now another quick question is that, you know, the more that I'm learning about VLM and we're talking about it, it sounds, and I'm not sure how familiar you are, but it sounds quasi similar to if you've heard of Nvidia's inference microservices or NIMs where it sounds kind of similar in terms of like VLM and Nvidia nims. I mean I can think of a couple of differences just based on this conversation what I know about the Nvidia product. But I would love to hear if you are familiar with Nvidia nims. How does VLM compare and contrast with Nvidia nims?
- [15:54]
B
I don't think they are competitors. I would say NIM is more like a product provided by Nvidia. So that's for the end user. But inside Nim it can use VLM as a serving engine. So I think VLM is kind of like the core engine and then NIM ships will to customers.
- [16:18]
A
For the longest time I kind of thought VLM was something that Nvidia, it was an Nvidia thing. But it's not like it's just within some of their stuff, right. To be able to serve the models, the containerized models in Docker, et cetera. So that makes kind of a lot of sense. So you know, on to kind of the, the next one, we kind of talked about the tech, we've kind of talked about everything else. Kind of the high level of what VLM does. But let's take it a step down, like next level down. You know, we talked about the model surveying, we talked kiwicache, we talked about all that. What is the next thing that you've seen people kind of do with vllm?
- [16:54]
B
I think the next thing would be we need to co evolve with hardware vendor and also model vendor. So by hardware vendor I mean like Nvidia amd, they are designing the next generation hardware to run these AI workloads efficiently. And by model vendors, I mean like people who train the models, not only open source one, but also private ones. Like how they design the model, what's the next generation of model, how do they look like? So these basically shaped the development of vlm.
- [17:34]
A
So I know you know VLM is open source. So I question though is that how does, let's just say the way that anthropic develops a model versus OpenAI. How does that affect VLM if VLM is open source?
- [17:50]
B
Yeah, so VLM is open source and what people see is the support that we support open source models. Right? So for Qin for Deep seq, the models are already out there so people can see how WHEELM supports it. But actually for private models there are also many like VLM supports. It's just they are not open sourced. So we will discuss with lot of the model vendors even if they are not open source, like how VLM can support these models very well that actually they internally also use wlm.
- [18:26]
A
I'm kind of referencing back to your other point, right? It's like yes, you guys, you know, support the open source models and you do some private ones as well. But you were saying hey, one thing that we can get better at or you know, potentially changes with you know, getting more kind of out in front of the curve right of like with let's just use Deep Seek developing their, their next model or whatever, it doesn't matter or whoever and but how does. I guess my question is is that is the model's fundamentally different in the way like the model's fundamentally different in the way that they're Created that requires work on your end is one of the leading developers on VLM to have to go in and change your code base and how you do things. Because of the way that they kind of design and architect their model.
- [19:11]
B
That's actually a very big important part of our daily life. So open source models are still very popular nowadays and every day we have several open source models in our pipeline to be released that we need to support. So from an outsider perspective they might say like oh, they are just transformers. I mean they are the techniques that people say attention is all you need. But from a practical perspective every model looks very different and especially when people are trying to scale the models to longer context, to more intelligence, to more modalities. So we are seeing a great variation of the model architecture and this is also coupled with the innovation of the hardware advances. So like Nvidia designing their chips more and more aggressively across like similar to the domain specific architecture. So these chips are designed mainly for this workload which means you cannot get the general purpose applications. The diversity in model architecture and also the diversity in the hardware architecture brings the major work we need to do.
- [20:32]
A
So model architecture, hardware architecture is what keeps you up at night working basically changes at it. Okay, I mean that makes sense. You know, obviously kind of being an open source project, you want to work with everyone that you possibly can, but sometimes there's just not, you know, enough hours in the day. I mean you don't have to share anything if it's private, but you know, kind of what's on the roadmap for VLM and say the next year, is it bringing on to work with other pieces of silicon, Is it, you know, preparing for a new set of models like anything you could share would be
- [21:04]
B
awesome, but you don't have to as I mentioned. So VLM kind of sits in the interaction between model vendor and also hardware vendor. So what we do basically is supporting new models on new hardware. So Nvidia has this grace backwell that's already announced a while ago and it's massively out. And there are also next generation like Guara Rubin. So yeah, basically we need to work with Nvidia to make sure that VLM runs on the, that runs great on the newest powerful hardware. And also there are model vendors that trying to pursue that better intelligence and more with more efficiency. So we also need to work with model vendors to see what's the efficient architecture that can for example scale the context trends to 1 million tokens while still keeping the level intelligence.
- [22:05]
A
So last couple of questions And I'll go ahead and wrap up, you know, with vlm. You know, we, you kind of talked about coding, we talked about Clawbot, we've talked about a whole bunch. But you know, can you name and you don't. If you know company names, you don't have to share that. But you know where VLM is being used. Do you have like say two interesting use cases that you could share with everyone listening now that maybe gets their wheels spinning on ideas.
- [22:31]
B
Companies using Willem, the list can be very, very long. Say for example the cloudbot. I think Kimi from Moonshot supports the cloudbot and Moonshot also uses William. And Kun from China also uses Will Amp. And as I mentioned, AWS uses WILM to run its ruffles. And like, yeah, the name goes to very, very long. Like LinkedIn and Google and Apple. Big companies like who buys GPUs, runs inference. I think you can find VLM there.
- [23:09]
A
Okay, so basically anywhere they've got a host of GPUs, you might not know that you're seeing it, but you're going to. VLM is there, basically silent but deadly running in the background. I love that.
- [23:20]
B
Yeah, yeah. Because we support new models and yeah, I think this is kind of like what, how William is different from traditional software. Say for example, if you want to run a database or if you want to run a website, if the functionality, you are satisfied with the functionality, then you can just stay with the version. Right. But for vom, when new model comes out, they do not work with previous vom, so that's a major motivation for people to upgrade and keep up with us to use the newest model.
- [23:57]
A
Okay. So there's not historical support necessarily. It's like once the new model comes out, you're on to the next.
- [24:03]
B
Yeah, we usually focus on the current, the main branch and we only focus on that. Yeah. And if people are using the old version, we encourage them to upgrade.
- [24:13]
A
So Kai Cha, this has been a great episode, extremely educational. We're going to wrap it up. But before you go, what I like to do to end it is do two things. The first one is pretend someone just started watching this episode, like right this second, you know, give the 32nd minute elevator pitch on VLM. What is it, what does it do and why they should be downloading it from GitHub right now.
- [24:38]
B
I would say this is more for like data center scale. If you are running large language model on GPUs. GPUs are expensive and models are getting larger and complicated. So if you want to run the big intelligence model, then you should use VOM that we work with model vendors and hardware vendors to make sure it runs efficiently. It. It saves your cost, basically.
- [25:03]
A
Running many concurrent, you know, processes in the cloud saves you time, energy, effort. Check out vlm. And lastly, Kai Chow, where. If people want to follow you, you know, on social media, where, where can they find you?
- [25:17]
B
I'm available, like in GitHub, I think. Yeah, you can, you can search my name and GitHub. Yeah, I have an active GitHub profile. I'm also on Twitter. There were several like, Mimis in Twitter. That's like, mention me and then get 1 million page views. I don't know why people get so crazy.
- [25:37]
A
I don't know why people get so crazy. That's hilarious. That's funny.
- [25:41]
B
Yeah, but basically, yeah, you can search for my name. I'm available on like, LinkedIn, GitHub.
- [25:46]
A
Perfect. Okay. Amazing. Well, Kaicha, I really appreciate the time. And with that, you know, ends another great episode. I hope everyone learned a lot about vlm. I know that I did definitely check it out. Just type VLM into Google. It'll take you straight to GitHub, the research page. You can learn kind of everything about it.
- [26:03]
B
And also you can visit the GitHub page and then give us a star. I think we are approaching the 80,000 GitHub stars.
- [26:11]
A
All right, so do it. Go out and start the GitHub repo. So with that, this is Logan with reshaping workflows with Delpro Precision and Nvidia RTX GPUs. Hope you had a great time and we'll see you on the next one. This podcast was produced in partnership with Amaze Media Labs.