← Liftoff with Keith: Conversations with Super Founders and Growth Experts sharing useful insights
Liftoff with Keith: Conversations with Super Founders and Growth Experts sharing useful insights
The Untold Challenges of Scaling AI: Insights from Supermicro's Vik Malyala
00:08:39
Loading summary
Transcript17 lines
- [00:06]
A
Wow. So happy to introduce Vic Maliala, managing director from Supermicro. Vic, how are you today?
- [00:12]
B
Excellent, Kate, thank you. Thanks for having me here. Yeah.
- [00:15]
A
Great to have Supermicro here at the Infra AI 2025 event. How are things going over there?
- [00:21]
B
Super busy. In a very good way. Yes, in a very good way.
- [00:26]
A
Give us some stories, give us some color in terms of, I think when.
- [00:29]
B
The company primarily focusing on infrastruct, we never had to really think about how they get deployed before. You take the servers, it goes, it's in a data center, as long as you have a rack, you stick it in, you connect the power and ethernet cables and whatnot, then you are set. That was the case. No longer the case anymore. So what is happening is our engagement, not just with the technology providers such as Nvidia, AMD and Intel, in bringing that into a shape and form through our products. Most of the data centers aren't quite equipped to take advantage of the latest infrastructure in a proper way. I mean, you can always take a system and stick it in a rack and then you can use it. It's not going to be efficient. It may not be the optimized, but as the deployment sizes are getting bigger and bigger, if you don't run the infrastructure efficiently, then you are wasting a lot of resources. And part of it is that we are engaged with so many more data centers, our data center builders and data center customers, that I never thought I would actually do that in the past. So while it's extra work, it's quite exciting. And this is solving a really good problem. Right. So that's what keeps us super busy.
- [01:49]
A
Yeah. What keeps me up at night thinking about how do I get access to all these power demands?
- [01:54]
B
Oh yeah. I mean, I kid you not, because what is happening is majority of the data centers, I mean if you take a look at the broad category, the ones that are throwing like billions and billions of dollars, I mean you have seen the Microsoft, the metas and whatnot, like 70, 80 billion a year on infrastructure spend. But not everyone have the deep enough pockets or the business case to justify that. Majority of them are going to be like the tier 2 cloud service providers. And some of them they have their own data center, some of them, they actually are going to take data center space as a lease to go build it. So there is a huge power requirement and there is a problem for sure. But this is where again it goes back to how we understand the data center environment and make sure that we bring the optimized platforms into that so within a given power profile, how do you fit the maximum infrastructure to run your problems? Yeah.
- [02:51]
A
And how do you keep it safe? The 200 kilowatt power.
- [02:54]
B
You know what, this is the funny part, right? Like, you know, when I started working on this, I said like 200 kilowatts is a good number. So let's talk about it, right? But then the way I was thinking was that 200 kilowatts that is already happening today because majority of the deployments that people are seeing, you know, in general in the air cooled environment, they typically put like four systems per rack, right? That's like 40 to 50 kilowatts. But the efficient data centers and the ones that have a large cluster deployment, we are already clocking at 200 kilowatts today. We are not talking about future. But then if you think about what will happen in the very near future, I would say within a year to year and a half timeframe, 500 kilowatts to a megawatt per rack is not going to be surprising to people. But how are they going to be ready for that if they do not know how the technology is evolving to kind of have a requirement for that to fit in their data center? So this is where I think 200kW is exciting to start with. But it is just like a comma. I mean, we have several more things to go before it actually hits the limits of power per rack.
- [04:08]
A
So in your conversations with customers, you're talking to them about these power challenges, these safety concerns. How are you coaching them in terms of balance, balancing everything from the density and the golden.
- [04:19]
B
You know what, it's not so much about coaching, but it's like working towards solving a problem. So what we have seen is that whenever these kind of deployments happen, we go and talk to the data center, the customer or the data center. Think of like almost auditing the data center auditing. In a sense, when we roll in the racks, are they going to fit in easily? Like for example, when you start packing, let's say eight or even 12 systems, this is like 2,000 to 3,000 pounds. Or the tiles are strong enough to basically take the weight, either a rolling weight or a static weight. Once you start putting these things in place, when you start powering, do they have enough power drop? Is it like a 63amps power drop or is it 100amp power drop? How many of them are needed, whether they even have the ability to do that, that kind of thing. So what we do is we kind of do an audit RSI survey, we call it. And based on that, we know these are the capabilities and limitations of the data center. And then we know what are the capabilities and the limitations of the systems. So then we say like, okay, based on this, let's say for example, you are doing a Superpower two type of deployment. Then we say like, okay, maybe you should do four systems or maybe you can do eight systems. Or by the way, you have liquid cooling since you have enough power and enough cooling and all these things. I can actually pack this much because every bit of it is all about optimizing. So let's say if I'm able to pack all of them in a very small form factor, then the latencies are going to be less and then the cost of the network infrastructure is going to be lowered. Ultimately, people need to solve a problem while spending the less that they can, the least amount of money. This is where the discussions are happening. It's not so much about teaching or preaching, but it's about us working together to solve a problem that the customers have, getting the maximum amount of efficient infrastructure. So the amount of resources that they have is optimized.
- [06:21]
A
Yeah. Let me take one last question with you, Vic. I have the head of Supermicro division. Looking forward about a year or two, what are you most excited about from an innovation standpoint?
- [06:33]
B
I think again, keeping AI umbrella in mind here today, much of the deployments are happening on training and the training of the clusters. The size of the clusters is getting bigger and bigger and it's going to happen. So it kind of forces us to bring more efficient platforms. So today we're talking about 75 to 85% of the heat to be rejected using liquid cooling. You know, if I, if I think about like in the very near future, it may be like even 95 or 98% of the heat rejected, which basically means the power that you have is going to be going for infrastructure, which kind of makes us work hard to bring more efficient platforms. That's one thing that keeps us excited because we are solving a real world problem. Because the sustainability is one of the key things, especially with data centers and AI infrastructure, we are going to make it as less impactful in a negative way for the customers. How do I reset? Let me say it again. The data centers nowadays are basically taking like 2 to 3% of the total energy. So we want to make sure that the carbon footprint is reduced, that the only way to do that is bring more efficient platforms. So this is one thing that keeps us exciting. The second aspect of it is the inference platforms. Ultimately, all this training and model development and everything need to translate into making our lives better and how do we do that? So the services are going to evolve and people are going to take this AI infrastructure to the edge, maybe the client devices or whatnot. All of it requires again, things moving or at least expanding from a core to all the way to the edge. So we are seeing end to end story developing in the next couple of years, which keeps us super excited.
- [08:25]
A
Vik, this is fantastic. I'm excited for Supermicro and yourself. Although it looks like you'll be a little busy.
- [08:32]
B
I hope to stay busy and solve real world problems.
- [08:36]
A
Thank you, thank you.