B (4:52)

Awesome. I really, really like the logo. I mean, I like every logo of every release that comes out. But this is like so much creativity with the Norse mythology, the magical kind of, you know, aspect it has, and also the kind of representation of the different Personas, I guess, of people involved in every release. Right. It's awesome. So I guess my next question is like, this is your first time being a release lead, if I'm not mistaken, Right?

C (5:25)

Yeah. I think typically people, you know, they have the opportunity to come through and be the release lead once. We have like a sub project owner in SIG release that oversees every version and supports the release lead. But yeah, this is my first time leading the release. I shadowed the release lead once. So every. Every release lead gets three to five shadows to support them. So I had an opportunity of doing that once. And then I also led two of the sub teams. The release lead oversees multiple sub teams. Right now it's down to four we consolidated some of them, but we have the enhancements. Sub team Release communications, release documentation and Release Signal. Release Signal doing the bug triage and CI signal triage.

B (6:15)

And so how was your experience in general? How did you find it?

C (6:20)

Yeah, so I've been on the release team for three years. There were two cycles I took off. But yeah, it's been a really fulfilling journey getting to bounce around some of the different sub teams. I started with CI Signal at the time before it was merged with Bug Triage into Release Signal. That's where I started on version 127 and that was definitely a pretty busy sub team to start with because I had to dig into a bunch of CI builds that were set up all around the Kubernetes project. And the Kubernetes project is organized in special interest groups, so there's a bunch of different groups that own different tests that are testing their features. And throughout the development and implementation, bug fixing and whatnot, we would have to monitor CI builds to make sure that the project continues to be stable. And once we run into issues, we'd have to correlate with GitHub and Slack and reach out to the owners and make sure things were fixed right away. That was a lot of work. There was a great fulfilling learning curve to it. From there I continued to work on release docs and other areas of the release team. So yeah, I've got a lot of experience. I eventually started to lead some of these sub teams and like I said, shadow the release lead. So yeah, it's been really exciting to see like this project and like all the guardrails and all the people that come together and just like this whole process that we have set up on delivering the second biggest open source project in the world, it was a great perspective and learning opportunity.

B (8:17)

Nice. I do have a question just out of curiosity and I'm going off script here a little bit. Do you work with Kubernetes day to day as part of your main job?

C (8:29)

Yeah, I have. Right now I've been more focused on AWS without Kubernetes just acutely what I'm doing, being assigned at work. But I've worked with Kubernetes platforms at my last two companies at, you know, in the K12 education and Department of Defense roles that I had. Now I'm in medical devices and I just started this job earlier this year after a layoff and getting a new job. Fortunately found something. But right now they have. We have like a legacy environment in AWS and I'm looking for Opportunities to bring Kubernetes in. But. But yeah, I do have some exposure to it as a user. Yeah, so.

B (9:16)

So the question that, the follow up question then is, is like having experience, hands on working with Kubernetes, does that help with the role of a release lead?

C (9:25)

Oh yeah, totally. Because I think one of the things that the release team is most is most influential in is setting the quality gates for users end users, making sure that throughout all the contributors in the project, we're hitting our guardrails with making sure we have all of our criteria for the Kubernetes enhancement proposals, making sure that we have stability with that release signal process, and making sure that all the new features are well communicated and the documentation is very comprehensive and easy to follow, and also making sure all the user facing changes are documented. I think it has been really helpful to be a user for Kubernetes because then I can kind of come in with that perspective and empathy of like, okay, what really matters to me as a user, what am I personally excited about to see and what is it going to be like for the folks after we release this thing? So that perspective definitely helps.

B (10:33)

Awesome. I asked this question mostly because I was wondering how much of your time do you spend really diving in versus how much you spend just coordinating? My previous interviews, there's quite a lot of work that goes into coordinating between the different teams as a release lead. And do you find your time kind of geeking out to try to figure out what's wrong or having to like say, remind yourself, maybe that's not worth my time? Like, how was your experience?

C (11:02)

Yeah, I will say, like, definitely. Coordination, I think is the biggest thing that the release lead takes on. I had a wonderful team of people to work with throughout this release. Both like the shadows that were supporting me, the leads that were supporting me on SIG release as well, that are consistently there, release to release, and then also the sub project or the sub team leads with, like I was saying, communications, signal enhancements and docs, they were all really great. And most of the time when I did start geeking out and digging deep, I would find that my team was doing excellent work. I really had nothing to worry about and my trust was in good hands. So a lot of what I did was delegation and, you know, kind of wrangling cats around the project, if you will, making sure everything's going on the timeline and it's going smooth. And even just having the shadows to back me up, they would also. And being in different time zones, people would pick up on things even before I did. So a lot of it was being an arbiter, making decisions and yeah, coordinating things. But again, as an end user, I also was really excited to see what the project was doing and where it was going.

B (12:23)

Awesome. Well, I want to spend a little bit of time diving into this new version. I'm going to start here with some stats and then we can go a little bit into details. This is 60 enhancements, so that's 17 features going to stable, 19 going to beta and 22 going to alpha. And then there is a bunch of deprecations roughly around from what I can see here, like four or five I think. And the major one obviously is the NGINX ingress retirement which will be coming up like the, like the project will switch to best maintenance efforts in 2026 and then it will be fully gone in the future. So let's talk about the new stuff. What can you highlight? What's your favorite? What are the things that excite you?

C (13:18)

Yeah, there's a lot of really great features that I'm excited about here. I think I'll start with our Blockbuster feature. This is just something I'm excited about. It's the first thing that we have mentioned on our announcement blog post and it enables smarter scaling for Kubernetes clusters. That's the in place pod resource updates. That is KEP 1287 going into stable. And what's really powerful about this is that we can make in place updates to adjust the cpu, memory requests and limits on the pod without restarting it. It's all updated on the fly. And that's really big because as a DevOps engineer that has worked with Kubernetes clusters, I'd have to go in and modify those things and completely flip the pod, which would cause an interruption for whatever application or service I have running on it. I think this is really huge as we're running a consistent workload that needs to dynamically adjust itself and vertically scale up that we have that capability to update it without any outage. So I'm really excited about that. A lot of good work was put into that. We also have great new features for the scheduling of workloads. We introduced game scheduling support, so that introduces a new workload object that will group a bunch of pods together. So when they're scheduled, they're either scheduled all at once or not at all. Which I think is really powerful for AI workloads that might have a bunch of training training jobs that are coordinating with the same stateful data set in Cases like that, there's definitely a lot of coordination and dependencies there. So it's good to have Kubernetes finally be able to do that natively without having to have third party tools that have to coordinate all of those pods being up at the same time. We also have extended toleration taint and toleration for threshold placements. So we can use numerical comparisons to determine what nodes a pod will get scheduled on. So that could be really big. Say for instance, you have on demand nodes or spot instant nodes. You can maybe give a scoring system to that. And then at scheduling time the pod can determine, okay, I really need something that's reliable and stable, so I'm going to do on demand rather than a spot instance. There's node declare features, so nodes can have a lot of different versions or underlining operating systems on their hosts. And depending on that, we'll determine what sort of Kubernetes features those nodes can support. So we finally have this capability for nodes to declare the features that they do support at schedule time. And then lastly we have opportunistic batching, which is really great for making some of the same scheduling decisions for pods and jobs that are very similar to one another so they can get scheduled much faster when it's time to be scheduled again. That's not even getting into the security updates that we have with pod certifications. Building a better impersonation model so that machines won't join the cluster and pull sensitive information from pods introducing user namespaces. So any pods that require elevation to root access will have their own user ID that they're running on the machine. So they're not elevating to like full root on the host. Yeah, just a lot of really great features I can go on and on about.

B (17:34)

It's actually interesting to me that a lot of these features that I've been following very closely because they all sort of have their own unique characteristic and unique use cases are falling together in this particular release. Right. It's. It's things that have been in the making for a while. You mentioned the in placepod update that we've been talking about that for a very long time. And that's to me, it's not an end goal by its own. It's just like the beginning of a much, much more interesting set of features in the future, like CPU boosting. A lot of interesting things that you could do, especially for AI workloads. I didn't know about the, the, the, the node. What was the feature you talked about like the node node declared features. Yeah, the node declared feature. That's very interesting. That's like. So the node will be able to say I support this and this workload and then the. That sounds like taints and tolerations but done better in a way.

C (18:33)

Yeah. Actually just to clarify and folks that want to look into this as well, it's kept 5328 and so this is the actual feature gates that Kubernetes has. A node could be set up to declare what features it supports. I think this is pretty powerful both for AI and for edge computing. For AI, maybe there are nodes that have a capable GPU that can support those training instances. Then it can declare that it has that available for Edge. You might have a data center that's on the edge that has a lot of different machines that are being used and it can be kind of hard to manage those with a limited ingress and egress. So being able to label what features that each of those nodes support, I think is going to be really powerful for these advanced use cases.

B (19:33)

Yeah, it's a super interesting feature. I was skipping through the the blog and I think among all of this stuff you talked about, one of my favorite features is the POD certificate for workload identity. So basically being able to supply an identity to POD in an automated way that have been I think one of the biggest headaches for doing multi tenancy on Kubernetes for a while, right?

C (20:01)

Oh yeah, totally. And there are a lot of third party tools that you would bring in. Maybe Spire or Cert Manager or whatever. That's something I'm just really excited about with 1.35 is to see the simplification that we're doing with cluster architectures, making some of these features native so that we don't have to pull in these complex tools.

B (20:24)

Yeah. And another one that I was just literally looking at today is being able to use images as a storage source. So instead of for AI workloads, this is like a huge problem. Right. Like AI, like these large language models are huge and you have to somehow make them available to the, to the, to the POD as a volume. But now being able to like base the, the volume itself on an actual image that you pull from an OCI registry is super, super cool.

C (20:54)

Yeah, I kept 40, 46, 39 OCI artifact as a volume going into beta. Yeah, this is really exciting for edge computing as well. To be able to attach a data set as a container image. I think that's going to be really big for the packaging of the workloads that you would do as you roll it out, maybe you have some removable media that you're using to install those workloads on the edge. So yeah, I'm really excited about that as well.

B (21:29)

Yeah. And also being able to package both your application and the data it needs as a single format so you don't have to package them as two different formats. Right. So pretty cool. You're talking a lot about edge. Are you like using or did you have experience working with edge use cases for Kubernetes?

C (21:45)

Yeah, at my prior job in Department of Defense, I worked for a company called Defense Unicorns. It was a startup working on, working on the. Basically the continuous delivery of workloads in kubernetes in air gapped environments. So things like a submarine or like a fighter jet. So yeah, the Department of Defense has like a lot of, has a lot of great use cases for deploying things in an edge environment. So I learned a lot in that role and learned enough about how they do things while I was there to be really excited about edge computing.

B (22:30)

Nice. Yeah, like working for a cloud provider, I don't really get an opportunity to work on use cases with edge computing. So it sounds like something super interesting actually.

C (22:39)

Yeah. I also like earlier in my career worked on a point of sale system, but that was before I was getting really deep in Kubernetes. But I've read these articles and heard these stories of places like fast food or retail that are deploying kubernetes clusters on site and managing everything. Like you'd go and check out your shopping and it'd be running through a Kubernetes cluster that's completely on site. So that's another awesome use case for Edge.

B (23:11)

Yeah. To my knowledge, I think the first company to come out with these use cases they blogged about it was Chick Fil a in the US and then it was followed by like McDonald's and then a bunch of other companies and then it suddenly opened up this thing. Oh yeah. We could actually deploy kubernetes on the edge or in a submarine or in fighter jet or whatever. Right?

C (23:31)

Yeah. I mean I think even like Target, like some, some retail companies are running their point of sale through Kubernetes and yeah, it's, it's very exciting stuff.

B (23:41)

Awesome. We're going to make sure to leave some links for people who want to read about this stuff if they want. I mean this is awesome. I really love how you dived into some of these features. Anything else you want to talk about? Deprecation of NGINX ingress in 2026. Wink, wink.

C (24:03)

I know that's a really big topic. It's not, you know, it's not strictly a part of version 1:35. I'm happy to comment on it though, because as someone who has been a Kubernetes user for a long time, that's something I've used for a while. I think it served us all well. I think it really brings up a good point about the sustainability of open source, that we need the active maintainers available to support these things. And I think with nginx Ingress we saw that it's just hard to continue safely without enough of that maintainer community. I'm really excited though, because the community is working towards long term sustainability and maintainability around other features like the Gateway API, which will be really powerful for tackling some of the same use cases. So anybody that is on NGINX Ingress, I recommend checking out the Gateway API.

B (25:13)

Yeah, it also brings up a very interesting point about the fact that the maturity of the community means that people care about simplicity and then at some point it's just not worth it to continue spending time on something that is unmaintainable and it's just like better to just stop working on it and move on.

C (25:32)

Yeah, it's definitely, yeah, it's definitely a tricky thing but you know, that's, that's kind of the, the beast of open source. And you know, this Kubernetes just being like such a big project, what's in like, you know, there being like the CNCF landscape largely revolving around it and everything is that there's, you know, there's, there's so many, there's so many solutions that are out there and you know, we wanted to give time for folks to investigate those and transition off.

B (26:06)

Awesome. Well Drew, thank you so much. This was really insightful. I hope you will have some time to relax over the holidays.

C (26:14)

Me too. Yeah.

B (26:16)

Anything else you want to add? Anything you want to tell our listeners?

C (26:22)

Yeah, I just wanted to give a heads up if you're an operator. I wanted to call out some of the deprecations that we have. We have removed support for cgroups v1, so your host OS on your nodes will have to support cgroups v2. We also are deprecating IPVs. That's another thing that's been burdened to maintain. So I Suggest switching to nftables and then this release is the last call for containerd1.x. We've transition to 2x. A long time ago, I think it was version 125, but now we're completely removing it in 136. So I thought it would be prudent for me to give a shout out to those things so that, you know, we can. We can all check to make sure that we're ready to run this next great version of Kubernetes and unlock these good capabilities.

B (27:23)

I mean, definitely, that's. These are very good things to shout out because we all know that not everybody is at the edge of upgrading to the latest and greatest version. So, yeah, the more you delay it, the more this thing. I mean, the more you delay it, the more it's gonna come to hunt you at some point.

C (27:39)

So to realize, yeah, I've. I've been in roles before where it's like, wait a minute, we're not.

A (27:44)

We're.

C (27:44)

We're end of support. Oh, we're end of life. Oh, great. So, yeah, no, I feel it. I wanted to call out these things just to make sure. Just to make sure it's on your radar.

B (27:55)

Awesome. Well, thank you so much, Drew.

C (27:58)

Yeah, thanks for having me. It's been a blast to geek out about this stuff.

B (28:03)

Awesome. Thank you for coming on the show. I hope you will take some time to relax and chill over the holidays. I hope that our listeners will have time to do the same. And this is, as I said, gonna be the last episode of this year, so we're gonna also take some time off. The episode will come out next week and then next week, as in we're Friday, so next week somewhere. And yeah, we'll see you all in 2026.

C (28:28)

All right, sounds good. Happy holidays, everyone.

B (28:30)

Happy holidays.

A (28:34)

That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media @Kubernetes Pod or reach us by email @Kubernetes Podcastoogle.com you can also check out the website at KubernetesPodcast.com where you'll find transcripts, show notes and links. To subscribe, please consider rating us in your podcast player so we can help more people find and enjoy the show. Thanks for listening and we'll see you next time.