← Kubernetes Podcast from Google
Kubernetes Podcast from Google
Kubernetes Resource Orchestrator (KRO), with Jesse Butler and Nic Slattery
00:44:14
Today we welcome and to talk about the Kubernetes Resource Orchestrator, or KRO. Jesse works as a principal product manager at AWS and Nic is a Product Manager at Google. The Kubernetes Resource Orchestrator is a new cloud agnostic tool meant...
Loading summary
Transcript124 lines
- [00:00]
Abdel Sighiwar
Hi and welcome to the Kubernetes podcast from Google. I'm your host, Abdel Sighiwar.
- [00:04]
Kazlyn Fields
And I'm Kazlyn Fields.
- [00:16]
Abdel Sighiwar
In this episode we talk to Jesse Butler and Nick Slattery. Jesse is a Principal Product Manager at AWS and Nick is a Product Manager at Google. We talked about the Kubernetes Resource Orchestrator, or CRO for short. CRO is a new cloud agnostic tool meant to simplify Kubernetes resources grouping for developers and platforms Admin.
- [00:36]
Kazlyn Fields
But first, let's get to the news. Kubernetes introduced jobsets, an open source API for representing distributed jobs. Jobsets allows users to manage child jobs as a single unit using a higher level API. It also integrates with Q and allows users to submit Jobsets via Q data to oversubscribe their clusters by queuing workloads to run as capacity becomes available.
- [01:03]
Abdel Sighiwar
Google Cloud Next will be held in Las Vegas, Nevada on April 9 to 11. The Kubernetes podcast team will be on site, so come by and say hello if you are going to be there. Keep an eye on our social media for more info about where we'll be at the event.
- [01:17]
Kazlyn Fields
CAGENT is a new open source agent framework introduced by Solo. The framework, which is built on Autogen from Microsoft, allows DevOps and platform engineers to run AI agents in Kubernetes automating complex operations and troubleshooting tasks. And that's the news.
- [01:36]
Abdel Sighiwar
Hi everyone. Today we welcome Jesse Butler and Nick Slattery. Jesse works as a principal Product Manager at AWS and Nick is a product manager at Google. We will be talking about a recent join announcement called Kero or CRO. The Kubernetes Resource Orchestrator is a new cloud agnostic tool meant to simplify Kubernetes resource management for developers and platform admins. Welcome to the show, Jesse and Nick.
- [01:59]
Jesse Butler
Thanks so much for having us. Really, really happy to be here.
- [02:02]
Nick Slattery
Yeah, thank you.
- [02:03]
Abdel Sighiwar
I'm so excited for this episode. We've been trying hard to make it happen, so I'm excited that it's finally happening.
- [02:09]
Nick Slattery
Me too.
- [02:10]
Jesse Butler
Yeah, me as well. I think that was my fault, right? I had a bit of a plumbing issue in my house. So, you know.
- [02:14]
Abdel Sighiwar
Yeah, I am well aware of that. No problems. No problems.
- [02:18]
Nick Slattery
I'm not sure people want to hear about your plumbing issues, Jesse.
- [02:21]
Jesse Butler
Not mine, my house. So, yeah, back to the program.
- [02:24]
Abdel Sighiwar
Yes. All right, so I'm going to start with. I mean, I've been talking about Kero for a while. I've been posting about it. We wrote some Articles. We're still writing more articles. I just did literally a talk today at like 9am in the morning. I did like a one hour talk in an internal conference and Karo was the one that got the most amount of questions, which was super interesting. So let's start with an introduction. Can any of you, one of you or both of you tell us what is Kero?
- [02:50]
Nick Slattery
Yeah, sure. Well, I think the explanation requires a little bit of a backstory, but basically what we found in the the Google team is that most of our customers are building internal developer platforms on Kubernetes and they were struggling with the orchestration layer. So how to build an orchestration layer that is native to Kubernetes and works across clouds or across any of their third party tools. And what we found when we investigated this is that there was no way to group resources in Kubernetes in a way that it runs server side and it's Kubernetes native. And it just makes it easier for platform teams to build services that can be easily consumed by their end users, who are typically developers or data scientists. So CROW provides that mechanism to group resources and so platform teams can use CROW to essentially define repeatable patterns for how services should be consumed across their organizations.
- [03:48]
Jesse Butler
Yeah, that's exactly right. And I guess my backstory isn't I may be preloaded that. So we at AWS were having the same discussions, the same problems. No Kubernetes native feature that does this. We also were finding that customers were kind of writing their own custom controllers with custom resource definitions and kind of going through the cube builder journey, which is, it's fun and it's fine, but if you manage 50 or 100 of these things and it's not really your day job, it's not as fun. So that was another kind of problem we were trying to solve. It's, you know, when you're building Kubernetes operators simply to use other Kubernetes operators and not putting business logic into it, we're like, well, we could try to solve that problem. So yeah, like sort of last year, earlier last year we were working on this problem. Some open source friends at Azure pinged us, you know, said, we're trying to look at this problem and we know that you're looking at this problem. Maybe we could collaborate through a mutual customer. Both Nick and I heard that we were both collaborating. So we said, hey, let's just get in a room.
- [04:54]
Abdel Sighiwar
Yeah, right.
- [04:54]
Jesse Butler
And so we got in a room and it basically was a great set of conversations where we realized, hey, we all have the same problems. And you know, from the position of cloud, you know, service providers, we want to solve these problems for our customers. And one of the other problems in Kubernetes, as we all know, is that there's 85 different ways to do everything. So what we wanted to make sure is that we weren't going to build three new ideas that then customers had to choose and context switch between depending on where they're running our application. So we sort of said, well, let's just align. And so we took some of the ideas that Google was working on and some of the ideas that we had and some of the things that Azure was working on, and we combined them all kind of around this one simple primitive to make it, you know, ideally something that we could see as a Kubernetes feature rather than another end to end solution, something that you have to build and manage and learn all of the ins and outs of. Let's just say let's keep it simple and let's just make it sort of what it is. Limit the scope, don't try to boil the ocean. And that's what we did. And so yeah, it's been going really well. It's not, you know, viral, which makes me happy because, you know, we didn't want it to just explode and then because it's, you know, self actively pre alpha as a project.
- [06:09]
Abdel Sighiwar
Yeah.
- [06:10]
Jesse Butler
So right now what we're doing is kind of iterating toward an alpha, iterating toward code complete. As we're hearing from customers that we're on the right track, like customers also ecosystem members, community members, college kids who are hobbyists in Kubernetes. Like everybody really likes this idea. So that's kind of in a nutshell where we are and why we're on this journey.
- [06:32]
Abdel Sighiwar
I mean, you're saying you didn't want it to be viral and for a project that didn't want to be viral, I was pinging Nick yesterday. We were mentioned on the info queue. There is an article about it on the news stack. There is a long discussion on Reddit that you both have been responding to. So it's created some noise. Right? So there are some discussions, which is good.
- [06:52]
Jesse Butler
Yeah, it is good.
- [06:53]
Nick Slattery
Jesse's just being humble.
- [06:55]
Jesse Butler
Well, I mean, I think we're all pretty humble because we, you know, like I said, we just got in a room and said, hey, what if we solve this problem this way? And it just has resonated. And I think one of the really exciting things is that it's not yet another direction to go in. It's more returning to basics and saying, how does Kubernetes work? What can we do? And so that was sort of at the root of it. We said, well, let's not go build another huge ecosystem, let's not build a new dsl, let's not have a whole new tool chain, let's just look at Kubernetes. And so the heart of this project is just that, it's how do we make the KRM work for you, especially when you're just combining resources. Right. So at the heart of it we have this idea that we'll dynamically create, register, install and register crds for you on the fly based on your specification of this resource graph definition. And then we have a single controller that adapts and it's an adaptive controller. So in place it learns how to support those new resources. So we think that it really simplifies that whole model. And certainly if you just want to use resource A and resource B together, you shouldn't have to write a whole new controller. So that's kind of part of it. And then the other part is as we do see a lot of interest in it, we're really trying to keep it focused. We just want this project to be this project. We don't necessarily need it to be the next new thing. It's really just Kubernetes. It's just looking at it a little differently.
- [08:18]
Abdel Sighiwar
Yeah. So I want to step back a little bit and try to answer a very specific question. Because Kero has some stuff that makes it different from tools that already exist that sort of try to solve the same problem. But I think that resource grouping as you discussed, or resource composition or however you want to call it, why do you think that that's important? Why in your mind being able to group resources under an umbrella resource or a meta resource is important?
- [08:45]
Nick Slattery
I can give the high level answer. I think Jesse, you can get more in the technical details if you like. But Jesse kind of already spoke to two of our goals were simplicity. So how do we solve this problem in the simplest possible way using whatever tools are already available to us? And then focus. So focus the scope of the project so it doesn't snowball into this behemoth or some end to end solution that we're pitching. It's just focused on a particular problem. And that problem is platform teams who run applications on Kubernetes and they're tasked with providing a Kubernetes based platform to their end users and a bunch of other services on top of that, and then they have to go and sell that platform to their end users as well, which means they need to provide a simple end user experience. So that's the use case for the customers that we're talking to and people we're addressing with CRO. And the challenge they were having is that they wanted to provide that simple end user experience, but they also wanted the monitoring capabilities of Kubernetes. So Kubernetes provides this thing called CRDS that lets you extend the Kubernetes API. And so we thought, okay, let's just use CRDs, let's not invent a new construct, let's make everything native to kubernetes so that the professionals working on Kubernetes can use all the benefits of it, but also then provide a developer who may not know what Kubernetes is with an experience that they can wrap their heads around. So that's kind of. I know that's a vague answer to your question, Abdel, but it is very focused on the actual problem that we're trying to solve. So what are our customers saying to us? What do they actually want us to do? I think our team at Google and Jesse's team at Amazon and our counterparts at Azure wouldn't have all come together around this unless all of us had customers saying, we want you to do exactly this, please. So that's really the impetus for the whole project.
- [10:43]
Jesse Butler
I actually like your answer because it's not vague, right? It's pretty specific. This is a problem we have and honestly it's across the ecosystem. If you are a hobbyist, you'll inherit this, right? I want to get really deep into Kubernetes and the next thing you know, it's like you. It's like the meme of the woman standing with the 55 ethernet cables going, help. You know, I just want email. It's kind of like that. So I like that answer. I think the complexity or even like the vagueness comes from how you get there, right? There's just so many different ways to get here and a lot of them come with other sort of whole ecosystems and sort of this other tool chains and other solutions. And sort of like in Kubernetes we kind of get back to that. Like that question that has been driving a lot of us for several years is like, if Kubernetes is a platform for platforms, then where is the platform? It's like we keep having these incredibly powerful abstractions and Kubernetes just got so much stuff, right? How do we leverage that with the next layer of abstraction? We have an enormous amount of awesome projects and tools and capabilities in the CNCF ecosystem, and those work with Kubernetes often as it is. And really, that's where we said, hey, if we try to get down to first principles and really look at the ground truths of the problems that we're trying to solve, we could build these little primitives that extend on other things that already existed, like Cell. Right, let's use Cell. Let's make better use of it. So maybe we can avoid more DSLs or more having yet another compiler. Compiler sort of stuff. So I think that's why that answer really works is that it sounds vague, but it's just a big, broad problem space. And so that's really the space that we're playing in and trying to just solve one little problem that's hopefully useful for a lot of use cases and other projects to integrate with.
- [12:31]
Nick Slattery
Yeah, I agree with that.
- [12:33]
Abdel Sighiwar
And I think in a way, the core of the problem that Karo is trying to solve, creating these abstraction layers is not new. Like, there have been tools trying to do that. I mean, arguably Knative was trying to do that specifically for serving applications, but that was kind of the purpose, in a very dumb way, if you want. Right. Like, I'm not trying to compare the two, but I think that's one question that comes. I mean, this is basically on the Reddit thread that I've been reading. How do you, if you would have to explain to somebody how Carol is different than what we have now, and what we have now are things like Terraform, which I think is easy to understand, but like Helm and kpt, like, why would I use Carol versus what I already use? Arguably Helm, because Helm is server side, technically.
- [13:13]
Jesse Butler
Yeah, I think there's two questions there. I'd love to start with the Terraform question. And then maybe because Nick and I have both talked about all of this before, so maybe then we'll switch over to how it differs from other Kubernetes space. So. So I think, you know, Terraform is amazing. Terraform allows you to build modules that abstract abstractions, and you can do all of these incredibly powerful things. There's a couple of promises with Terraform that once you start using it, you realize, well, okay, that was a good idea. Right? So, for example, portability. And we see this with Kubernetes as well, right? Yes, the tool is the same no matter where you run the, you know, wherever your environment is. But everybody knows that if you're on AWS and you're on Google with the same application, with the same types of resources, 80% of your code's different. Right. And that's where modules come in. You can just ingest these things and now you simplify your life. So that model really works. And it worked with Puppet and Chef. It worked with Salt and Ansible. It worked way back in the day with bash grips and duct tape. Right. It really is a pattern that we have just reproduced repeatedly. And this is just sort of like how do we bring that into kubernetes? And I think that's where we differ with this project because it's not really. This project is different from Terraform. Kubernetes is different from using terraform on a MacBook to drive infrastructure as code. So leaning into sort of saying people use Kubernetes because it is an open standard, because it's an evolving de facto open standard. So we use it and we don't then want to context switch out of it to do things like provisioning Kubernetes clusters or managing add ons. Right. So I think that's the big difference of like prior other tools and coming into Kubernetes that does open you up to KPT and Helm and Crossplane and all these other things that are also Kubernetes.
- [15:00]
Abdel Sighiwar
Yeah.
- [15:00]
Jesse Butler
So maybe Segue Nick, you can talk about that a little bit.
- [15:04]
Abdel Sighiwar
Yeah. So then how is that? I mean the Terraform distinguishing is pretty easy to make. So then a helm KPT like Crossplane, where do we draw the line essentially?
- [15:14]
Nick Slattery
Yeah, well, I'll address the HELM question. I guess basically we wouldn't be interested in investing in something new if there weren't a gap. And like Jesse said, I think CROW is probably better defined by the things that we're not doing. We are not building a new templating language. We are not designing a new way to extend Kubernetes. We're just using CRDs and we're using existing templating languages like Cell. We had a proof of concept in Google that used Ginger as well, which usually makes people cringe when I say that I love Ginger.
- [15:48]
Abdel Sighiwar
I'm one of the weirdos.
- [15:51]
Nick Slattery
But then if you look at an existing tool like Helm and then look at the use case that we are targeting, which again is platform teams building essentially custom APIs on Kubernetes for consumption by some end user like a developer. Helm wasn't really designed for that use case.
- [16:07]
Jesse Butler
Right.
- [16:08]
Nick Slattery
And again, this is getting into my personal opinion. I'm sure some people agree and Some disagree. Helm is really good for distributing third party software. Like, if I was going to start a company today and I needed to distribute my software on kubernetes, I would definitely package it as a helm chart. There's not much thought needs to go into that. It's very good for that use case. It is brittle, I suppose, and complicated. And what we hear from customers is that it's hard to manage the lifecycle of helm charts and it's hard to track dependencies. When you apply HELM to the use case of building a platform for your internal developer consumption, it's a complicated use case. It often requires customization, depending on the region that you're deploying a service in or the team that's consuming it, things like that. So you end up with these layers and layers of helm charts, often stored in repos, which then have different layers of permissions within a repo. And so then when something goes wrong, trying to troubleshoot and figure out which layer that problem is coming from is a huge challenge.
- [17:09]
Abdel Sighiwar
Yeah.
- [17:10]
Nick Slattery
Right. So. And this applies to many other tools out there. I'm using Helm as an example, but the feedback was it would be really nice if I could just come to my cluster to see the intended state and the actual state of all my individual resources, but then also how they are meant to be grouped together. And if there's things like dependency ordering, I want to be able to see that in the cluster.
- [17:33]
Abdel Sighiwar
Yeah, Right.
- [17:34]
Nick Slattery
Because I'm a platform admin, I'm a Kubernetes professional. That's where I want to see my source of truth.
- [17:40]
Abdel Sighiwar
Yeah. And if I may add to that one maybe simple addition, I think that, like what you were talking about, if I am in a platform admin trying to sell an IDP to developers, telling them to use something that already exists in the cluster, like resource graph definition versus telling them, hey, there is this helm chart here. Go install this cli. To be able to use this helm chart. There is a huge difference between something that already exists and it's easy to use with the tools you have, or something that you have to adopt, kind of. Right. So you mentioned something very interesting about the resource ordering or the dependency graph or dags. Right. I did some tests, as in I deployed the sample example, which is deployment to a service with ingress, and it just worked out of the box. Perfect. Right. Can you talk a little bit about where do you see that dependency graph becoming interesting as people use and adopt Kera going forward? Right. Why do we even need it?
- [18:31]
Jesse Butler
Yeah, I think with that simple sample application. You see why you need it, right? Because what you want to do is configure some resources against the values that exist in other resources. And so that's one of the basic things that CROW does, is that when it looks at a resource graph definition that you give it, it automatically infers that dependency by config injection. Right? You know, if you need a foo and you need a bar and bar needs to be configured with the, you know, the UUID of the foo, you need the foo first, right? And that's pretty easy to do. What's not super easy to do, which is some ongoing conversations that we're having now. Because one of the primary use cases here is beyond simple web applications. We want platform engineers and small DevOps teams alike to be able to kind of author their own cloud building blocks. We want them to be able to say, oh, give me a Kubernetes cluster and have all of the complexity of VPCs and subnets and security profiles and all that stuff just abstracted away. We're going to do that. You need to understand the dependency management there. And yes, again, if you have OS subnet and it needs, you need a vpc. So you have this cloud thing and you have, okay, cool, I can configure injection, right? Will tell me. Configuration injection will tell me the order there. What if that isn't part of it? What if you're just saying give me a foo and give me a bar and inherently I need the foo before the bar. But I haven't done any configuration mapping. That's a kind of an external data source that doesn't yet exist, but we've left a door open for. And I think what's interesting there, it gives you the ability to plug this into other kind of state aware configuration management solutions. As one of the really important things here is if you're building a primitive that should help everybody move the ball forward, you got to make sure you have the things that they need to actually integrate. So that's a really good question. Big picture. I think typically the use cases that we're seeing, and we already do have some users of the project, customers of ours and other folks that we've seen in the community building fairly complex things. And that is at its root the ability to say, well, okay, I need some infrastructure and then I'm going to have a couple of services and they're going to run on that infrastructure and then I'm going to have my application which leverages those services. So there's some configuration elements that I need out of them. You just put it in the RGD specification using that simple schema, which is another piece of the project that's really interesting to us is that, as Nick said, make it simple. Right. Nice. Human, readable. Feels very Kubernetes. Ish. You just define that rgd, define how these resources interact and CRO sorts the rest of it out. Right. Uses a dag. We have static analysis built into it so that if you've done something that won't work, will tell you right up front that doesn't work. Looking like kind of further down the line at other verification methodologies that we can integrate. So you can have like sort of system verification capabilities that leverage these primitives. That's sort of the big picture. But yeah, I think it's super important and vital. That whole resource we abstractly call resource orchestration. Right. Having all this bundle of stuff, how does it work? You don't need to worry about it.
- [21:42]
Abdel Sighiwar
Yeah.
- [21:43]
Nick Slattery
Can I add, because Jesse just made me think of something because your answer was great. But it gets to how flexible CROW is because it is built as native to Kubernetes. Jesse just described a few use cases that require some ordering of resources. There's simple use cases like I need a service account created before my deployment, which is just using Kubernetes native objects. But then that same workflow applies to much more complicated use cases. And Jesse started talking about subnets and other resources that are not native to Kubernetes, which then gets into how CROW works with other kubernetes controllers. And I think Jesse was alluding to ack@AWS and there's KCC at Google. And these tools allow you to use CROW to orchestrate not just Kubernetes objects, but any of your cloud resources. So in theory, because CRO is native to Kubernetes, any third party vendor can build a Kubernetes controller that orchestrates their API. And then it would be compatible with CROW as well.
- [22:48]
Abdel Sighiwar
Yeah, and so that's a really good segue to my next question because it's very important to stress the fact that CRO works with Kubernetes objects so they could be native, which are the stuff that you have inside Kubernetes already, or any technically object that you install through a crd, kcc, ack, aso, whatever. Right. So before we get there, I think I'm gonna make a very, very stupid comparison. And please feel free to tell me you are dumb here. Right. Looking at how you define a resource graph definition in crow, which for the people who are gonna listen to us, that's how you define a new crd. Technically. Putting it very simply, it feels to me like you are defining an API interface because in the schema you're exposing the inputs or the parameters that the user will be able to define. And then in the resources section you are hiding away the complexity of the underlying resources. Some of the values in those underlying resources will be coming from what the user defines and some will be set to default values. Is that like a fair way of describing it?
- [23:53]
Nick Slattery
Absolutely, yeah, I think that's fair. Yeah. It's essentially a custom API that is native to Kubernetes. Yeah.
- [23:58]
Abdel Sighiwar
Cool, good. I'm happy about it. So in that case, where do you see that becoming important for like cloud specific resources, like defining an EKS or a GKE cluster? I am quite sure that eks, similar to gke, has tons of parameters, Right. Maybe hundreds or maybe less, I don't know. But like this is the point, like how do you see that as an advantage that Karo has?
- [24:24]
Jesse Butler
Yeah. So I'll just loop back a moment. I did blow right, by how you would create those VPCs and subnets. So thank you Nick for keeping me grounded and for any listener who's. How did you do that? Crow provisions VPCs. No, you use it with other controllers, right?
- [24:39]
Abdel Sighiwar
Yes.
- [24:39]
Jesse Butler
And so like ack, aso, kcc, these sort of solutions to integrate to SDKs in the cloud to get cloud resources. So yeah, you basically are saying, I'm going to create this. So for example, we'll use cluster, it could be GKE cluster, it could be EKs, AKs, whatever. So if you have a cluster, maybe. I don't think EKs clusters have thousands of settings. We have a fair amount, but there's a lot of resources under the covers and those resources all have settings, some of them are substantial. So at the end of the day, if you have this single RGD that is cluster, it's probably a dozen or so resources each with dozens of configuration elements. So it gets wildly complex. So one of the most important things in defining this as sort of, to use your framing, right, to use this as an API, to say I want to expose an API to my. For example, I'm a platform engineer, I want to expose an API to my developers for self service. Cluster provisioning should be something that we all have. That's a great goal to have, right. So a developer comes along, says, give me a cluster, there's a dozen resources under there that I don't want them actually even touching or necessarily out of the context of a cluster. And then there's a bunch of resource configuration for each one of them that they have code to write, they have business logic to implement, they don't want to worry about that. One of the cool things here with the rgd, in defining that API, you get to define, as we had said, the resources, their ordering, their dependencies, how configuration is shared among them, what new resources have values that get injected as configuration to other elements in this one thing called a cluster. So I give that to you, and I thought you now have to set what, 140, 150 configuration elements? No, thank you. Right. So what we do with the RGD is we also allow you to expose only the things that you want your resource consumer to have to worry about. And so to your point, the implementation of this API, almost like the backend, is a collection of resources and you can set default configurations and you can inject configuration from other resources. So you basically now are in this position to say, okay, with this cluster example, give me a vpc, some subnets, some security groups, some, you know, some IM resources, whatever. Put this all together, create the cluster with them. So now you're creating resources with something like KCC or ack. You're coming through, creating these resources, injecting configuration where you need to, setting default configuration where you want to, and at the end of the day, you're exposing it through a new CRD that CROW will dynamically create, install and register in the cluster for you, where your downstream developer just sets like name and region and that's it. Yeah, right, they just set those two things because in the back end of that API. Right, I did air quotes there for podcast listeners. The back end of the API is actually like an API. So you're implementing a lot of complexity and relationship management and stuff in the back end, and you just expose the simplest view that you see fit. And of course, for power users, maybe you have two flavors, maybe they can set a handful of configuration variables that give them a little bit more control. So this is entirely in your hands as far as authoring a resource graph definition. So, yeah, that's my take. I think that's a really good question.
- [28:03]
Nick Slattery
Can I throw in another example? Yeah, because your question's really good, which is why is this important? Or why does this make life better for managing cloud resources outside of Kubernetes? And it again gets to, how do we make life easy for a platform engineer who has to sell something to their end users? So we need to make the end user's life easier, easy as well. And from a platform engineer's point of view, it's okay. I have this Kubernetes platform because my developers are running their applications on Kubernetes and I've built all these GitOps pipelines that are native to Kubernetes. Oh, and hey, now I have the tools to use those same GitOps pipelines and the same monitoring tools with any cloud resources, whether it's related to Kubernetes or not. So if my end users, a developer needs a storage bucket, I can offer them that service as something that's native to kubernetes. The developer doesn't need to know about Kubernetes, by the way. They just, like Jesse said, input the name of a storage bucket and the region. They don't need to know anything about all these GitOps pipelines and all of the platform engineering stuff. But from a platform engineering point of view, it's okay. Now my developers can provision a storage bucket and the way that that gets provisioned is compatible with all the other tools that I use to run their Kubernetes based applications.
- [29:15]
Abdel Sighiwar
Yeah, yeah. And if I might add there, as you folks were talking, it just made me think of something. If you have written even the simplest thing like the deployment service ingress in the past, one of the most annoying things is that sometimes there is the same value that you have to repeat everywhere. Like in a deployment, it's the same value in the name, in the template, in the label is the same thing in the service, same thing in the ingress. If you can just set it once on the RGD definition and you don't have to worry about it. Like it's just get trickled down through a variable and I'm putting air quotes here as a variable and then it just works, right?
- [29:48]
Jesse Butler
Yeah, there are nice little syntactic things like that in the sticky behavior of using Crow every day, which I do. You know, I create wildly scalable hello world applications all the time and it's sort of like having a favorite programming language. Like you don't think about UX or like human design when you think, oh, I love Go. Oh, I really like JavaScript but that plays a big role. So just these little things that we've found through kind of, you know, approaching the problem from first principles and then building up what we think would work, we have little things like that. So I think there's a third use case that is one of my favorites which is like maybe I'm just a one person shop selling like WordPress. Right?
- [30:31]
Abdel Sighiwar
Oh yeah.
- [30:31]
Jesse Butler
I can simplify my life by using Kubernetes. Because we know that, right? Like we can. It's much easier to do that than running, you know, big monoliths on compute VMs. Right. So I'm using Kubernetes even if I don't have a downstream consumer, even if it's just for myself. I can take all of the complexity and all of the configuration things and I'm always typing the same defaults and these, but sometimes they're a little different. So I have to write sort of the same string 85 times a day and just bake it into an RGD. And now I just create them on the fly and I have a little bash script that runs and I just make money. Right.
- [31:07]
Nick Slattery
Yay.
- [31:07]
Jesse Butler
So like that is sort of. There's so much of this where it's just, if you look at it just like this little primitive building block to build other building blocks. It's very powerful and very exciting.
- [31:17]
Abdel Sighiwar
Yeah. I love how it always comes back to bash at some point eventually. Right.
- [31:20]
Jesse Butler
So for me it always does. It used to be corn shell, but I'm, you know. Yeah, I finally transitioned.
- [31:26]
Abdel Sighiwar
Okay, welcome to the right side. It's awesome. Well folks, this was a really insightful discussion. What's next? What's coming for Carol? Where do you want people to check it out? What do you want people to go to do with it?
- [31:39]
Jesse Butler
Yeah, yeah. Come join us. One of the reasons we made a lot of noise about it is that we want everybody to know this is a super, super open project we think is a really exciting future for it. And it is fully vendor agnostic and we're sort of adopting cncf like governance just right out the gates. Just because that's what people are familiar with in this ecosystem. So super open meritocracy. Everybody's welcome. We have a community meeting every other week on Wednesday mornings, 9 Eastern Time. It is tomorrow. So if folks are interested or actually wait, I don't know when this podcast goes out.
- [32:15]
Abdel Sighiwar
No, this is not going to come live tomorrow.
- [32:17]
Jesse Butler
It's March, March 5th, so in the past there was one. But every other week from there also. Yeah, we have the Crow channel in the Kubernetes Slack. You can come join us. Obviously. GitHub.com Crow Run. Crow is the project. Please come take a look. Open issues file prs really happy to have you contribute and join and just join the conversation.
- [32:40]
Nick Slattery
Yeah, I'll just say that there's a lot of reporting that typically opens with Google and Amazon and Microsoft are collaborating on this thing, which has been true, but we don't want to own it. It's a community project. Anyone is welcome to contribute. We do have a handful of independent contributors already, which is great. But yeah, by no means is this just a Google, Amazon and Microsoft thing.
- [33:03]
Abdel Sighiwar
Yeah, it's very important. The project is completely independent. And yeah, as long as you guys continue working and making good work, I'll continue advocating for it. So you have my back in there.
- [33:12]
Nick Slattery
We appreciate that.
- [33:14]
Abdel Sighiwar
Thank you very much, Jesse and Nick, thanks for your time.
- [33:16]
Jesse Butler
Thanks, Abdel, thanks so much. Thanks for having us.
- [33:18]
Abdel Sighiwar
Yeah, thank you. Have a good one.
- [33:20]
Nick Slattery
You too.
- [33:22]
Kazlyn Fields
Thank you for that interview, Abdel. I've been looking forward to learning about KRO or Kubernetes Resource Orchestrator. That's what it stands for, right?
- [33:31]
Abdel Sighiwar
Correct.
- [33:31]
Kazlyn Fields
But I missed it because I was out, so you're going to have to tell me all about it.
- [33:36]
Abdel Sighiwar
Oh yeah. So it's quite interesting how this came to exist because it was literally a quick discussion with Nick last year at kubecon North America and Salt Lake City, and up to that time I did not know about scale at all. I had no idea that this project existed. And then we started chatting and then I realized, oh snap, this is quite big. And then I started digging into it. Yeah. So essentially, in a nutshell, as you might have heard in the interview, it is not a Kubernetes native for now, because you still need to install the crd, but as natively as it can be in Kubernetes for you to do resource grouping. So you can make higher level abstractions of complex Kubernetes objects and make them available to developers as a simple object with a simple schema on quote unquote, as in like simple fields. So you can do complex deployments, you can hide. Like if you have a super complex object with lots of options, you can set some options to default values and only expose what the developers are allowed to change. Things like that. So you can think about it as resource grouping, essentially Meta. Pretty much, yeah.
- [34:41]
Kazlyn Fields
So it's taking existing native Kubernetes resources and it's a CRD on top of them that allows you to group them into deployable units.
- [34:51]
Abdel Sighiwar
That's the beauty of KRO native and not native. Right? As far as Kubernetes is concerned, the object could be native to Kubernetes, so the underlying objects could be native to Kubernetes like deployment, service, whatever, but they could also be External cloud resources. Right.
- [35:06]
Kazlyn Fields
And the goal is to give you a deployable unit. So this is something that you define.
- [35:10]
Abdel Sighiwar
The goal is to give the platform administrator a way to group resources and make them easy for developers to deploy. So there are like kind of two Personas.
- [35:21]
Kazlyn Fields
What about the management side of it? I assume that it makes coordinating management of the components easier. Correct.
- [35:27]
Abdel Sighiwar
So there are two Personas. Like as a platform admin, you would define those like abstractions. Right. Using Kero, using this thing called Resource graph definition rgd. And then in your resource graph definition you have two sections. The section that says this is how the objects will appear to the end users. The name of the object, the schema, the fields that the user is supposed to define. And then once the user creates an object of this type, these are the objects that will be created under the hood.
- [35:54]
Kazlyn Fields
Interesting. I mean, custom resource definitions could be kind of like that too. Right.
- [36:00]
Abdel Sighiwar
And that's the beauty of krl, because it will register this as a crd, so you don't need to write a CRD from scratch.
- [36:06]
Kazlyn Fields
Interesting.
- [36:08]
Abdel Sighiwar
And then the operator, so the component that will act on those objects and create them or delete them or manage them. That essentially depends if the objects are native to Kubernetes. So it's the existing operators in Kubernetes. If it's a third party cloud object, then you will have to install external operators to be able to manage these objects.
- [36:26]
Kazlyn Fields
Right, Right.
- [36:27]
Abdel Sighiwar
So Kero is quite agnostic in that sense.
- [36:29]
Kazlyn Fields
And I do want to mention also that this is a pattern that I'm seeing more and more is extensibility in Kubernetes is becoming more and more something outside of Kubernetes itself. Kubernetes itself is really big. If we let it get too much bigger, it's just going to be unw. The process for getting something into mainline Kubernetes at this point takes so long. And so the community is leaning more and more toward these types of custom resource definition extensibility, types of features, rather than built into Kubernetes features.
- [37:05]
Abdel Sighiwar
And this is also kind of the philosophy of KRO is keep things simple. So it's like borrowing from a book of Kubernetes of keeping things simple. So it kind of sits between I want to build a CRD and an operator, but I don't want to jump right away to do it. So let me try if Karo can solve this problem with me before I start writing, like crafting. So it's not going to solve all the use cases of people needing to write their own custom resource definition and operators, but it might solve part of it. So you can think about it as like solving the lower common denominator between the use cases for CRDs and operators. And then if you need to do something more complex, you still can go the route of building your own thing. Right.
- [37:47]
Kazlyn Fields
I feel like we usually talk about CRDs and operators kind of in the same breath because people don't think that much about the implementation of the thing within kubernetes, the operator versus the implementation of what you want to run. Like the definition of it, the custom resource definition.
- [38:04]
Abdel Sighiwar
So in my head, I might be wrong. Right. In my head, a CRD is sort of useless. Like a CRD is by definition a definition. Right. So a definition without something to act on it is pretty much useless.
- [38:19]
Kazlyn Fields
Exactly. But people use the terms interchangeably.
- [38:22]
Abdel Sighiwar
Correct.
- [38:23]
Kazlyn Fields
Most of the time.
- [38:25]
Abdel Sighiwar
And I think that's because most of the time a CRD has an accompanying operator that goes along with it. Right. To make that CRD useful. Or it should. Yeah, that's a very good point.
- [38:35]
Kazlyn Fields
Yes. If it doesn't, then it's really useless.
- [38:39]
Abdel Sighiwar
Yes, pretty much.
- [38:40]
Kazlyn Fields
But it's a definition of something that doesn't exist in kubernetes.
- [38:44]
Jesse Butler
Exactly.
- [38:45]
Abdel Sighiwar
And so, yeah, no, it's pretty cool. The discussion have been pretty illuminating. I've been like, I helped write one of the blog posts. I did some talks about it. I did a webinar last week about it. But I think like talking to the people, like the people who are driving the projects on AWS and Google. And by the way, keep in mind that this is between Google, AWS and Azure. So it's not like one cloud provider effort. It's like a common effort between the major cloud providers.
- [39:09]
Kazlyn Fields
Yeah, that's very interesting. I noticed when the blog post came out that it was about it being a collaborative effort between all of the major cloud providers. And I was like, is it really? And then as I've seen more and more stuff about it, it's like, yeah, it really is.
- [39:24]
Abdel Sighiwar
Because all of them have been thinking about doing the same thing and I think at some point they just realized they're trying to solve the same problem. So why not just solve it together?
- [39:33]
Kazlyn Fields
And this allowed you to talk to someone. Awesome. Jesse Butler. I can't believe we gotten 10 minutes into the chatter without me mentioning I to be on the same team as Jesse Butler at a previous job. And so I'm extremely jealous that you got to talk with him.
- [39:48]
Abdel Sighiwar
Yeah, he seems cool. We didn't meet in person, we just talked online. But I like the way he has this very easy way of describing and explaining things. It's pretty cool.
- [39:58]
Kazlyn Fields
He is very good at that and very deep too. He's had a long technical career in the Linux space and really knows what he's talking about. I always enjoy learning from Jesse.
- [40:08]
Abdel Sighiwar
Nice. Awesome. Yeah, it was pretty cool. It was a very good conversation.
- [40:12]
Kazlyn Fields
Nice. Interesting. So one last question. Were there any particular use cases that you all discussed or that you want to imagine for this? I'd love an example.
- [40:22]
Abdel Sighiwar
So, yes, so in my talks, one of the very basic examples I give is the example of standard deployments. Imagine you have a web app you need to deploy, right? The minimum you have to create in a kubernetes is a deployment, a service and an ingress to make that thing access. Or maybe a Kubernetes and a service of type load balancer, depending what kind of load balancer you want.
- [40:42]
Kazlyn Fields
This is true.
- [40:43]
Abdel Sighiwar
And so if you go actually to KRO run, there is an example where they kind of use KRO to hide the complexity of all of this. So instead of having this long YAML file which contains your deployment, your service and your ingress, you create an rgd, a resource graph definition that hides that complexity. And then your application becomes a nine lines kind of thing instead of like, you know, whatever 20 or 30 or 40 line. And this is actually. So there was somebody publishing on the new stack a couple of articles about Kero and the example that they give is WordPress. So imagine you are an admin that just needs to deploy WordPress websites for your customers and essentially all deployments are the same except like few things changes between deployments. So using Kro you can just like templatize in a way that deployment and only change the things that are different between your customers between the different deployments you need to do.
- [41:38]
Kazlyn Fields
Interesting. Yeah, I have done that and I would like to have that. Excellent. I have something to do to be very fair.
- [41:47]
Abdel Sighiwar
I mean there is other tools that can do that. Like Helm is notoriously known for that.
- [41:51]
Kazlyn Fields
But I was wondering about that.
- [41:53]
Abdel Sighiwar
Yeah, Helm solves a completely different problem. There are multiple things and one of them being that HELM doesn't have reconciliation. Like you have to trigger HELM in case the deployments get screwed up and you need to redeploy it. You have to trigger HELM manually.
- [42:07]
Kazlyn Fields
It's more on the definition side than the operator side. Helm, it's like correct a library of things that you can deploy to your kubernetes cluster but it doesn't do anything to the cluster itself.
- [42:17]
Abdel Sighiwar
It's a packaging tool. Right. So KDL is more like on the server side. So you have that like a reconciliation loop as well. So if somebody goes and access, accidentally delete something that needs to be there, then you have the benefits of the conciliation loop of Kubernetes. And Kubernetes will just, you know, recreate the objects. And I'm not talking here about scale down a deployment or delete pods. Like, if somebody deletes the deployment altogether, kubernetes out of the box is not going to reconcile it. But because Karo is built as an operator, it will be reconciled.
- [42:44]
Kazlyn Fields
Nice. All right.
- [42:47]
Abdel Sighiwar
And so there is. If you go on Kero run, there are a bunch of examples. This is just one simple one. And then there is like more complex ones. There is one actually on the AWS side, which is using AKS to deploy an LLM. So it's super complex with a lot of objects and stuff. So if you are looking to contribute, looking to lend a hand, I think that the team will be very happy to have somebody helping out.
- [43:09]
Kazlyn Fields
Yeah. Join me in checking this out next time. All right. Thank you, Abdel. I'm glad that I got to learn about that and glad that you got to meet someone awesome in addition to the awesome people that you already know. It's what we do.
- [43:25]
Abdel Sighiwar
Thank you.
- [43:27]
Kazlyn Fields
Thank you. That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media Kubernetes Pod or reach us by email@kubernetespodcastgoogle.com you can also check out the website@kubernetespodcast.com where you'll find transcripts, show notes and links. To subscribe, please consider writing us in your podcast player so we can help more people find and enjoy the show. Thanks for listening and we'll see you next time.