← Kubernetes Podcast from Google
Kubernetes Podcast from Google
Navigating AI Guidelines in Kubernetes, with Kat Cosgrove and Natali Vlatko
00:50:39
Loading summary
Transcript140 lines
- [00:00]
A
Hello and welcome to the kubernetes podcast from google. I'm your host kaslyn fields.
- [00:06]
B
And I am abdel sighwar.
- [00:17]
A
In this episode we talk with Kat Cosgrove and Natalie Vlatko. Kat and Natalie are both members of the Kubernetes Steering Committee and leads of SIGS release for CAT and Docs for Natalie. The Kubernetes Steering Committee recently published a new policy on AI use usage For Kubernetes contribution. We dive into that AI usage policy and explore the challenges and experiences AI has brought to Kubernetes maintainers.
- [00:42]
C
But first, let's get to the news.
- [00:47]
B
Apple has shipped version 1.0 of its native container tool for macOS. Built in Swift specifically for Apple Silicon, it departs from traditional shared VM setups like Docker Desktop by isolating every single Linux container inside its own dedicated micro VM using the native macOS virtualization framework,
- [01:05]
A
Google launched OpenRL, a new open source project designed to streamline the training and reinforcement learning loops of large language models. The tool brings declarative Kubernetes style resource orchestration concepts to the messy process of AI model fine tuning.
- [01:20]
B
At Kubecon Cloud Native Con India, the CNCF announced they added 14 new members, end users and nonprofit organizations. This highlights the continued growth of the cloud native ecosystem. One of the new members is Lovable, who was a recent guest on the show. We highly recommend you go listen to episode 268. About the Agent Sandbox Is a pod
- [01:40]
A
the right deployment unit for an AI agent? Lin sun from Solo published a community post on the CNCF blog questioning whether the classic Kubernetes pod primitive is still the best abstraction for hosting autonomous long running AI agents and introducing Agent Substrate, a project attempting to bring a solution to the table. And that's the news. Hello and welcome to the Kubernetes podcast from Google.
- [02:06]
C
I'm super excited today to be talking about AI in Kubernetes and whenever I say this people always think that I mean like running AI on Kubernetes or there's so many different angles that you can take for it. But today I want to talk about how AI is affecting the open source community. So for contributors, how are they using AI? What is the community thinking about, how AI is changing, how we do contribution, all that kind of stuff. And to join me to talk about that, I have Kat Cosgrove and Natalie Vladko. Would you like to introduce yourselves? Let's go with Kat first.
- [02:40]
D
Sure. During the day, my actual job, I am the Village Sorcerer at Village SQL, but in Kubernetes I am a technical lead for SigDocs. I own the release team sub project for Sig Release and I'm a member of the steering committee.
- [02:57]
C
Wonderful and welcome to the show. And Natalie, thanks for having me.
- [03:00]
E
In my day job I run the Open Source Program Office or OSPO for short at a little known company called Cisco which is a lot of fun. But in Kubernetes specifically I am one of the co chairs of sigdocs as well. And in terms of other open source fun, I'm also currently a steering committee member for the to do group which is a project of the Linux foundation that gets Open Source Program Office professionals and enthusiasts together to basically do good open source in the world.
- [03:31]
C
I'm so glad that I asked you to introduce yourself because I didn't know anything about that and now I do. I have not heard of the to do group.
- [03:38]
E
We are more than happy to have you in the community. You don't have to be like a member in order to join the Slack community that we have to ask questions or to use our resources. Happy to provide all of that later on.
- [03:49]
C
We'll include those links in the show notes if you would please send those over.
- [03:54]
E
Definitely.
- [03:54]
C
And both of you have had some experiences as very open source, involved folks with AI in open source and particularly in Kubernetes. Though if you have insights from other communities I'd be interested in those as well. But let's start off by diving right into it. There is actually a policy from Kubernetes about how AI is allowed to be used in the project. I think a lot of people probably don't know that. You should check out the policy. I will link it in the show notes. But with how fast everything is changing, all the different tools that people are trying to use and different ways that people are trying to use AI, I think it's really important to have some guidelines. And we've also heard interesting news stories about how AI is affecting Open Source. So let's start off by talking about what is in this relatively new it came out like a couple months ago relatively new AI policy for Kubernetes and Kat, you published it. So do you want to kick us off by giving us a little summary?
- [04:55]
D
I sure did publish it and it was less dramatic to publish that than I initially expected. I thought we would get a lot more pushback. But generally the vibe is you are welcome to use AI in the assistance of authoring a pr. That is fine, you can do that. We don't object to it, but you have to disclose it in the description of your pr. You cannot list the AI tooling as the co author. You can't co sign a commit with an AI tool. And that's a legal thing, right? Your AI tool cannot sign a cla, so it cannot co author a PR or cosign a commit. You also can't use the assisted by or co developed or whatever similar commit trailer for your disclosure. That is like sort of a marketing thing. Like we expect hyperscalers to behave. We don't expect necessarily that some smaller shady tools would behave. That's to prevent somebody from pointing at like the Kubernetes git history and saying, oh look, the Kubernetes project endorses our tool and slap our logo on their website. So put your AI disclosure in the PR description please. But also your commit messages can't be generated by AI. You need to write those yourself. And regular rules about character count and being descriptive apply when responding to review comments. You have to do that without relying on AI at all. We want to talk to you, the person. We want to talk to you about your code and make sure that you understand what you're submitting. And that includes maybe non native English. It's okay if English is not your first language and you think your English is bad. We would way rather handle that than talk to your LLM because then we can't like ensure that you actually understand what you're submitting. So please talk to us. We love talking to you the person. But otherwise you are responsible for your AI code.
- [07:06]
C
And that's kind of what it boils down to, this AI policy. We're not saying don't use AI. Sure it can cause some problems, which I think we'll get into in a little bit, but you can use AI. But there are some rules that we have to adhere to. And remember this is arguably the CNCF said it in like one thing that I can find that the Kubernetes project is the second largest open source project in the world. Regardless of that, this is a very, very large project with people all around the globe. So there's a lot of legal and kind of bureaucratic rules that we have to think about when we make policies and when we adopt new tools. And so I love the way that this policy worded because I think it acknowledges that without being like overly wordy and describing every little thing, it's like it sets out the rules and you can just kind of roll with that. But the first one that you mentioned That I want to point out especially is that we are not allowing folks to co author their commits with AI tools. You can use AI tools, but you can't specifically use the term co authoring. So you described that a little bit, but I want to reiterate that for folks because I think it's really important.
- [08:17]
D
Yeah. And when we say that, that means, like, you know how copilot can like commit directly to a PR and it shows up as a co author. Can't have that hard.
- [08:27]
C
No, like you said for legal reasons, because Copilot or whatever AI tool you're using cannot sign the. What does the cla actually stand for?
- [08:37]
D
Contributor License Agreement.
- [08:38]
C
Contributor License Agreement. There we go.
- [08:42]
E
That the reason that we say it cannot sign, not that it isn't able to technically sign it, because obviously it's a click on a button to verify your identity via email address and so on. But what it actually means in a legal sense is that your agent, your LLM, your, any kind of AI assisted tool isn't a legal person in the definition of the contributor License agreement. That's what that actually means. So we're not talking about the technicality of not being able to sign. We're talking about the legality of it's supposed to be a person that is human.
- [09:14]
C
And it also lends to the overarching idea of the policy of you need to understand what you're submitting. You as the human, are responsible for this code that's going into this open code base.
- [09:26]
E
Right. And I think the important part on that too, going on to the next area of why we want specifically you to disclose that you've used AI as part of the policy as well. We've definitely heard in the community since introducing the policy about how there is a possibility of, or maybe people are worried about bias coming in, that you must disclose if you used AI or not. And it is definitely linked to the idea that you're the human responsible for the contribution. And so us having the signal that you've used AI would possibly determine whether you're adhering to that accountability aspect of do you understand the contribution you're making or have you reviewed the contribution that's taking place? And being upfront to say that you've used AI helps us understand whether are we able to review this PR with you or not. Because people don't read policy quite often as well. And us asking once again in the PR to disclose, and I often, when I'm asking, say, can you please disclose if you've used AI so that we can review this PR better together. I give context as to why that question is asked so that people understand that it's not about me putting your review at the end of my pile because it's AI assisted, but it's more so to understand are you fully understanding of and have you fully reviewed the contribution that you're making?
- [10:49]
C
AI is a tool for improvement, not a replacement. And another really important aspect of this policy that relates to some experiences I've had in the open source community when I've talked with other co chairs, other maintainers of Kubernetes about AI. There was a meeting where I asked folks, are you seeing a huge influx of AI written low quality PRs that are overwhelming you? How's that going? And surprisingly to me, the maintainers actually said, meh, not really one because they were just as overwhelmed by the AI PRs as they were by the human PRs. Because we were already overwhelmed. We already had too few reviewers for the number of PRs coming in, so just as overwhelmed as before. But the biggest thing that they mentioned that they were frustrated by with the use of AI in prs was the comments because they were getting these AI written pages and pages long comments that told them nothing about the code that was actually being proposed and so they couldn't review the PRs properly. So to your point, Natalie, we want to review PRs with folks, right?
- [11:57]
E
Right.
- [11:57]
C
So do you want to share some more about that?
- [12:00]
E
Yeah, definitely. I've definitely come across PR that I reviewed that the AI tool that they've used have hallucinated an issue that is being fixed that doesn't exist. And I click on the issues that are referred to in PR descriptions to see what it's fixing. And I've come across 404s and it's just something that a couple of extra small bits of review, even if you are using that tooling to generate your PR description and so on, if you're not checking those things, it tells me that you care maybe a little bit less than I do about how this review process is going to go. And so I really want to make sure that I'm spending my time with folks who are also wanting to spend their time productively when they're contributing as well. Another interesting policy that we have on the doc side that also never gets read, but is absolutely heightened in contributions coming in because of AI is that we have a trivial edits policy where we say, hey, you want to fix a small spell check error, spelling error, should I say, in this specific document. And that's the entirety of your contribution. Have you looked through the rest of this document to see if there are other improvements that could be made? Because that's kind of like a trivial edit and maybe you can chuck that in with a couple of other improvements to make that review worthwhile time wise for your contribution and my input. And I will say that the amount of trivial edits we get is definitely on the rise now with the AI contributions. However, interestingly, that is where AI shines in terms of it being less error prone because the changes that are being presented are so small and so scoped that that actually isn't a horrible idea. Using AI as this advanced linter or documentation, for example, just like we would possibly use that in the same way as an advanced letter for reviewing a pr. So that's an interesting kind of like how to say something that supports the policy but also counters it at the same time because of the increase in the contributions coming by that tooling.
- [13:55]
C
Kind of a dichotomy.
- [13:57]
D
Docs gets hit harder by the AI stuff than other sigs do. DOX has a PR wrangler rotation, right? Which I have to do as well as a tech lead. And the number of PRs I review and then ultimately close because the actual content of the AI generated PR does not at all match. The content of the AI generated PR description is outrageous. And that like that takes time for me to find, right? Like that is completely time wasted. Those PRs get closed immediately. For me, that's not a conversation, that's just a hey, this is in violation of the AI use policy. Close immediately. No discussion. But I would rather be spending my time reviewing an actually useful contribution because it's unfortunate because sometimes the content of the PR is something that could be helpful. It is a genuine improvement to a piece of documentation. But the whole thing is AI generated. The PR description is AI generated and does not match the content. The comments from the user are AI generated. I at this point suspect that this is actually an agent that somebody has like logged into the account and signed the cla for it and turned it loose. And I have to spend 10 minutes figuring that out. And in that 10 minutes I could have cranked through two or three other actually useful VRs. So it's, it's obnoxious. It's a waste of time. The AI use policy doesn't actually slow those kinds of contributions down much, but it does give us something to easily point at that is written down as our public policy so that we don't have to fight about it. When we close the pr, which is a different kind of time saver, I
- [15:48]
C
guess that was really challenging in the early days of AI, where different folks within the community had different views about using AI tools and how they should be used and how it was affecting their specific area of the project. And there were some, some challenges on like, well, do I just straight out reject this? It is causing me problems, but it's just hard to tell if it's good or not. And I'm going to have to spend more time on it. I think your point about, like you're wasting all this time looking at these PRs that are not going to be high quality contributions to the project, also lends itself to a point that I like to tell new contributors about all the time. I like to say that good first issues are a trap because, like, everyone has to have a first issue. Like, it's important for us to note issues as good things for new contributors to do. But historically good first issues would get snapped up and then not completed. And then when they don't get completed, the reviewers have to go back and triage those again. And they're spending the time triaging those that they could be spending reviewing PRs that people are actually making and trying to get into the project. And AI kind of accelerates that in that now not only are people snapping up good first issues like their candy, they're also submitting something with them that is perhaps low quality, perhaps it's really wordy. It's going to take even more time for the Triagers to review that. Plus the volume of things that they have to review has increased. So it's kind of this vicious cycle that it creates for maintainers.
- [17:22]
E
Yeah, definitely. And that cycle, interestingly also comes from this area of, you know, folks wanting to use AI as this learning tool, which is great in some contexts. But Kubernetes as a project, in terms of how complex it can be, it's worth figuring out whether using it as a new to Kubernetes person, as in new to the community and new to contributing versus new to the technology overall and how to even learn what to do with it. I think that's also worth new contributors asking themselves. Because contributor open source generally with without AI has often been for new contributors, quite daunting. They often are not sure where to start. And we in our respective Sigs and Kaslin of contributor experience and Doc have a lot of really great guidelines and references of how to take that first step on the contributing side. But an AI can read those docs too and help out. So maybe applying the tool in a different way in order to make that first contribution is something folks can take away from this.
- [18:24]
C
I love when folks say that AI is an enhancer. If you understood something and are trying to create something with your knowledge, AI is going to enhance your ability to do that. If you do not know the thing and are trying to use AI to do the thing, it may just enhance your lack of knowledge of the thing and do things wrong and then you won't know that it's doing them wrong and then the reviewers have to catch that and that just spends more of their time.
- [18:52]
D
Yeah, and we're already kind of crispy, you know. But you know where is a great place to start if you want to contribute to Kubernetes or open source in general? It's your first time contributing to open source and you're understandably real nervous and you want somebody to hold your hand. Sig docs, Sig docs. You can come shadow one of us on a PR wrangler rotation and we will hold your hand for your first contribution to an open source project.
- [19:18]
E
Definitely. We also actually have issue wrangling as well, where we actually are trying to make sure we're creating better issues. I don't want to say a good first issue because I also don't believe that that's always a good thing, but being able to do issue triage where we're creating, or should I say fixing up issues that already exist to be ready for solving by a possible new contributor and them self assigning is also something that we try and do with folks who aren't necessarily new to kubernetes or docs, but they've been around for a little bit and feel like they want to contribute in some way that's beyond reviewing or sending in different kind of docs, amendments and stuff. So yeah, definitely, if you're interested in helping out in a way that's maybe a little different, but also helpful specifically to newer contributors, issue wrangling might be the way to go.
- [20:09]
C
And I want to dive a little bit deeper into that. I've been doing a whole bunch of workshops talking with folks in person. These have been for my day job Google workshops, but I always mention of course that I'm deeply involved in open source kubernetes and if you want to talk about kubernetes or open source contribution, I would love to talk with you about that. And I always get a whole a pretty big group of folks who are asking me about making their first Contributions, which I love. So if you wouldn't mind. And each of us are leads of SIGs that I think are in some ways particularly welcoming to newer folks. But I'd like to dive into that a little bit more. Docs has several different areas where it's pretty welcoming to new folks. You mentioned PR wrangling, but there are a few others. Do you want to give us a list?
- [20:54]
E
Natalie so yeah, firstly I think we often over index this idea of a first contribution, but in my opinion it's just getting involved and that is such a vague kind of thing depending on what is available or good for you in the way that you're working or how you like to possibly introduce yourself to a room of online strangers, which is kind of basically what open source is. And so for me I often first focus on getting involved by attending a meeting or lurking in a channel like that is the first, first step of what you should be doing. And then in those places is where you're going to find the links to the different resources that we have about how contributing actually takes place. You will often get folks that will come in and say, hi team, I am so and so and I'm so happy to be hopefully contributing to Kubernetes soon. Please guide me and let me know what you want me to do. We love the motivation, but we want you to have a bit more initiative in finding those resources yourself because we have placed them in areas that are very findable in all of the bookmark locations in our channels, in our areas. On our website they are constantly mentioned and referred to in our meetings and in our agenda documentation. And so attending a meeting or going into a channel to then be exposed to this information is step one in my opinion, every time as your first foray and contribution. But then on top of that how you then get involved. So Kat had already mentioned PR wrangling, which is great. All of us approvers in sigdocs do that on a weekly basis and we can always have one or two shadows that will help out. Issue wrangling is another place and specifically folks can get in contact with me. That is kind of my job in the sig to wrangle the wranglers on the issue side and then the other area to be able to kind of start getting involved on the SIGDOCS end. We also have the blog, we have a couple of blogs actually Kazlyn, you folks in Contrivex have a blog that you own and then Stigdocs also has a blog and I mention this because it also coincides specifically with a lot of work that happens on the release team side with the release communications and feature blogs that happen that come out of a release. We want blog reviewers. We want blog editors.
- [23:03]
A
Blog reviewers.
- [23:04]
C
Please, please.
- [23:06]
E
If you want to write a blog, that's great. We have a lot of blog writers, I will say, but what we don't have is writing buddies to team up with those writers to help them out. And that's actually something that we've recently introduced as a bit of a kind of like a rule of when you submit a blog. Also when you submit a feature blog as part of a Kubernetes release as well, where we team you up with a writing buddy who is often an author of a different blog and you review each other's work work so that you're helping out with the workload on the SigDocs blog team, which is also quite small and would love some more help.
- [23:41]
C
I love that approach to it of pairing up folks who are writing blogs to review each other's work. I said before and we'll say again, writing blogs, there's a lot of glory in that. You get your name on something, it's public. People love writing blogs. Reviewing blogs is one of those less visible things that is so, so important and so valued by the community. So we would really love to have folks who are willing to review blogs even if you're not. We've had folks come in who are not deeply technical, who do not maybe understand Kubernetes super deeply, but they can review English language writing. And so that can be super helpful in just getting different people's perspectives on how you would word something or if something comes across clearly. If you are someone who maybe English is not your first language, seeing if that writing makes sense to you, you are part of the target audience. So your review is still really, really valuable. So want to highlight the blog review opportunity for folks? We would love to have more folks.
- [24:47]
E
We actually also specifically have a style guide. And so even for folks who are not really sure how certain technical code, roadblocks or technical aspects of the blog should actually be formatted, our entire style guide actually addresses this. So learning our style guide, you're going to be able to review any single blog that comes in through the blog team of either Contrivex or sigdocs. So we can definitely also, in the specific notes for this podcast, share the style guide as well.
- [25:16]
C
Absolutely. And we bring that up in comms meetings a lot. We have a lot of folks who are interested in writing blogs. And we always encourage them to also review blogs and point them to SIGdocs resources on learning how to do that. And I love how you framed it of like your first thing should be just joining a meeting, lurking in a Slack channel when we do new contributor orientation. I always say those things are contributions. You don't usually think about it, but anything you do to support the community is a contribution that includes sitting around and just listening and trying to learn. That's one of the most valuable things that you can do to be part of the community. And then picking up something like PR wrangling or starting to dive into a specific area is kind of Day two and then I would say Release Team has one of the coolest, I would say Day three options. We can discuss why I think that But Kat, do you want to talk a little bit about Release Team and the opportunities available there?
- [26:10]
D
I am so happy to so first of all, I agree that the Release Team is more of a Day three thing. While we very much do have Release Team members who have never contributed to Kubernetes before or never contributed to open source before, it is the minority number of team members on any given release and it is harder to get selected that way. So the Kubernetes release cycle runs for roughly 14 weeks. It kind of varies depending on which release of the year it is. We do this three times a year and it is made up of four sub teams led by a release lead and their shadows. The sub teams you can apply for are Release Signal, which is the merge CI signal and Release Signal teams or bug triage teams, the Docs sub team, the Comm sub team and the Enhancement sub team. All of those teams are fairly technical. The most technical of them is really Signal. None of them require an intimate understanding of the internal workings of Kubernetes, which is why this is a good Day three thing. It is helpful if you understand like the lay of the land. You know what a SIG is, you know what a working group is, you know what a sub project is, you understand how GitHub works and how we're organized. That is helpful. Not a hard requirement. We do onboard all of the shadows at the beginning of the cycle and cover those things. But it is a wide open application process about two weeks before the beginning of any given cycle. So if you are contributing a little bit in other areas of the project or not, you are welcome to apply for the next release cycle. If you don't get in, which is likely, unfortunately we get around 2,250apportunants per release cycle for about 20 spots. Maybe 25.
- [28:04]
C
It's grown so much.
- [28:06]
D
Yeah, it's huge and it's competitive. That does not mean that you're bad. It doesn't mean your application sucks. It doesn't mean that we don't want you. It just means that we are blessed with an abundance of very, very good choices in the release team and you can still participate. Not being on the team does not mean that you can't help out with a release. All of our meetings are public. You can come to those meetings. We have the regular release team meeting on Wednesdays and then towards the end of the cycle we start doing burn down meetings daily. The Monday and Friday ones are on Zoom as regular. The Tuesday and Thursday ones are usually Async on Slack. But there will still be stuff you can do because we still never have enough people reviewing docs or blogs. Even with a fully staffed team of shadows, it's not enough people to handle the huge influx of documentation at the docs ready for review, freeze deadline or the feature blog deadline. We still need more than those like four people and whoever we can pull together from SIG Docs. So as a not team member, you are still welcome to help us review blogs and documentation. We can't like include you in the roster. We can't like assign work to you the way that we do shadows. But if you just hang around in the website repo, I promise you there will be so much to do and you are welcome to message SigDocs or me with my SIG relief hat on for some direction figuring out where to go. But yeah, it's a very popular and high visibility way to get org membership and start contributing to Kubernetes. But it is like drinking from a fire hose. You will be busy from day one for 14 weeks. So it is not the best choice in my opinion for a first contribution. It was my second contribution to Kubernetes
- [30:02]
C
and this is something that, you know, over 200 people applied to and if you get it, you need to be focused on it.
- [30:09]
D
Oh yeah, lock in.
- [30:10]
C
Yeah, CAT implemented some really important changes to the release shadow program that we talked about in your interview with Ubernettis, which I was going to try to check before this. I think it was our highest listened episode since Abdel and I took over this podcast. So people love Ubernettis.
- [30:29]
D
I'm so glad people love Ubernettis, but it is end of life now and you have got to upgrade. Please. Yeah, unless you're on like an LTS Thing with like Microsoft or whatever, it's end of life. Please upgrade off of Uber Netties.
- [30:44]
C
Yeah and so people love the structure of the release team, but Ubernetti's Natalie,
- [30:49]
E
I was about to say I wonder why people like it. Gee, is there a correlation between anime and tech somehow I'm not sure. I often like to like semi brag that I work on every release because I'm part of sigdocs and naturally help the Sig the release docs and release comms teams every cycle as my role of, you know, co chair of Sig Docs and reviewer of PRs and blogs. And so I think a lot of folks are able to to possibly claim that too if they want to maybe jump in and help out because they are interested in doing that or haven't even been able to get a spot on the release team. Yeah myself and well Kat obviously can brag about it given she's the sub project lead. But yeah, I work on every release. A small part of it, but still the reviewing part is just as important and we always need more folks.
- [31:35]
D
Yeah, I think no two sigs have more symbiotic relationship than release and docs. Release and architecture are like obviously heavily intertwined too, but we don't actually interact with them much throughout the cycle aside from PRR freeze and they kind of handle that entirely on their own. You know Sig docs and Sig Release
- [31:56]
C
have to work, but docs has its own freeze.
- [31:59]
D
Docs has its own freeze. That was not a hard freeze before I took over and now it is because I got tired of big docs having to bear the brunt of people not respecting soft deadlines for docs within the release. So it is a hard deadline with an exception process just like code freeze
- [32:19]
E
now and which we have tooling to help out to get some of your docs first time provisioned and put together and then the human can come in and review that after using this AI tool.
- [32:29]
C
Right.
- [32:30]
E
That's the part that's some of this tooling is possibly making easy. One other aspect I wanted to mention about the AI ness of kubernetes that is often happening that doesn't get talked about a lot, but it's actually the use of AI notetakers in meetings.
- [32:45]
C
Oh boy. I have opinions.
- [32:46]
E
Yes. The fact that all of us reacted in a certain way either with sounds or body language. As I said that we had a huge influx when they became a thing and it's dropped off little now. But I want to tell everyone out there listening that if you're not able to attend a meeting, I will kick out your AI note taking bot because we don't want it there.
- [33:07]
D
Oh yeah, same I report them to Zoom every time. Get out of here with that. The Fireflies AI things. Stop that. That's surveillance state behavior. I don't like it. We're not sending our robot assistants to open source meetings for us especially because.
- [33:22]
E
And people may not know this, but sick meetings are recorded or they should be by the folks moderating them and they're uploaded to YouTube playlists and they're available to watch later. So they are public meetings that if you're not able to attend, you can definitely watch over them later. You're able to read the agenda and the notes that are public for them too. But AI note taking in meetings is definitely something that's a no no. All of us different seek leads, chat in our own chairs and tech leads channel. We tell each other about the new Read AI talk or things like that that are coming in that we now need to kick and block. So heads up, don't send them, please.
- [33:57]
C
Yeah, we talk about that a lot in contributor experience because we own Zoom and a lot of how Kubernetes does its meetings and things. So we're working on some things there. Join some sync Contrpex meetings if you want to hear all the gory details about what we're thinking on that. But a good point there is that all the meetings are on YouTube. Feel free to use your AI to summarize the recording on YouTube and to review that yourself. But then once you've reviewed it and you're like, wow, I had something to say about that. Join the meeting to say it yourself.
- [34:29]
E
Yeah, definitely. It's also one of those things that can always be chatted about Async as well. I think folks, sometimes we talk about attending meetings and coming into the channels too. We know that we are not going to always have a perfect timed meeting for everyone that's available. So we work as Async obviously as possible too. You can always start discussions in our channels as well based on what you've heard in a meeting or watched in a recording after another place you can possibly contribute that likely needs more folks. Is a YouTube video not being uploaded quick enough. Maybe you can join the YouTube admins as part of Contribex to help out with that effort.
- [35:04]
C
Yeah, join us. I say us, but I mostly leave that work to the rest of the YouTube admin team. Yeah, one more thing that I want to come back to On AI in Kubernetes, I've found that when folks join Kubernetes these days, one question that I've gotten quite a bit that I find really funny is like, are we allowed to use AI here? Or is Kubernetes too old for that kind of thing or too bureaucratic for that kind of thing? And I hope we've answered a lot of that and you can see that you can use AI. We have guidelines around how we're using it. There are some blockages that we're working on with using AI tools because we have a very large project we've got to think about like privacy and security of everybody. So there's that. But follow the guidelines and things will be okay. But then as folks get into that day two, whether they're using AI tools or not, folks start asking me. My problem was that the reviewer didn't get to my PR in time.
- [36:05]
D
Oh, absolutely not.
- [36:06]
C
How are we going to use AI to improve the work of maintainers?
- [36:10]
D
Absolutely not even no, no. So here's the thing that I would love for new contributors to open source to really internalize. The overwhelming majority of us are working for free in our spare time for the love of the game. Natalie and I are very lucky and very unique in that this is part of our jobs. That's not true for most maintainers. It's certainly not true for most approvers or contributors. We're outliers, so we can afford to respond relatively quickly. The average contributor cannot. And nobody gets to demand that we work on anybody's time than our own. That applies to a random individual contributor as much as it applies to literally Google trying to exercise some strength to get us to move at a certain speed. So no, we will get to a review in the order it was received or in the order that we feel like it. You are not going to demand that we move any faster than that. It would be great if we had like some kind of tooling that allowed us to work more efficiently. Like some kind of smart AI powered pre review linter that does some of the like regular things that we all check every time that feel a little bit like toil. But the human labor involved in reviewing a PR is always going to happen on our own timescale and nobody else's. First, I'm going to only speak for myself on this and not the rest of the contributors or Natalie. But personally, if you found me like, if you send me a bunch of like kind of rude DMs trying to get me to review something more Quickly. I am going to wait as long as I can to review that as, like out of spite. Is that mean? Maybe. But so is sending me like six or seven DMs in a row because you tagged me in a PR and I didn't respond within 30 minutes. You know, like, I don't love that. It feels disrespectful. It's, it's super rude.
- [38:21]
C
I think there's a good reflection of the AI policy there. And number one, respect for humans, and number two, the human is still responsible. That's true. Whether you are submitting a PR or whether you are a reviewer responsible for approving a PR to go into the project. Maintainers take their responsibilities very, very seriously. And so they will put some human effort and thought into the work that they do, whether they're using AI tools or not. So that's very important to consider. That said, we are, like you said, interested in AI tools for maintainers that can help improve their processes. But when we talk about it as maintainers, we do take that kind of human centric view of the AI is a tool that's going to improve our workflows, not replace the maintainer in the process, which I think is very important for folks to hear. Natalie, do you want to add anything about AI and maintainership?
- [39:18]
E
I would like to add that you hadn't made a comment before that actually kind of made me laugh, Kazlyn, about folks asking is Kubernetes too old to have AI assistance in terms of contributions or so on. And I was just sitting there thinking, wow, is Kubernetes? Are we.
- [39:32]
D
We're geriatric?
- [39:34]
E
No, no, we are the millennials of open source.
- [39:37]
C
Because Linux are the boomers.
- [39:38]
D
Right?
- [39:39]
E
That's the Boomer project.
- [39:40]
D
And then you work on MySQL for God's sake.
- [39:45]
C
Oh man, MySQL is old.
- [39:47]
E
Yeah. And you've got like Kubernetes as the Millennial open source project, which it totally lines up with this idea of like, like, are we too old? And we're all sitting there like, freaking out like, oh my God, existential crisis. Is Kubernetes too old to get on board with new state of the art tech? And the answer is no. We're no. And asterisk. We have different ways that we need to work with some of this stuff. And part of it is because we are so large and so adopted, it would actually be super not great for us as maintainers and, and stewards of this project to bring in tooling that we didn't know was absolutely not only right for us as maintainers in terms of reviews and how contributions come in, but also didn't fit well along with the guidelines of the foundation that we're under. Which is exactly why I'm okay always saying no on the AI, you know, signed by XAI tool because of the legality of that.
- [40:40]
C
And it's.
- [40:41]
E
We're doing our due diligence and our open source due diluted for that as well. So I think that's another part to keep in consideration that there is always going to be interesting, fun, new state of the art technology in any of those boxes that folks that want to assess that they're always going to wonder, is Kubernetes getting, you know, on top of this or not? And the bottom line is that we are also a community of maintainers and people who get together and help to make those decisions. There's no misconcept to Kubernetes out there kind of making all the calls. It's a group of us who, as Kat mentioned, are largely doing this voluntarily. Even though I can do some of this for my job, a lot of this is after hours for me as well because my job is also very busy. And so we are trying to also keep up with these different kinds of technological trends and understand what's good for our project and make good decisions that will also help advance its, its maintainability and security.
- [41:37]
D
Yeah, we're not geriatric, we're preventative. Botox aged.
- [41:42]
E
I think you heard it here first, folks.
- [41:47]
D
I say that as somebody with fresh Botox. I just got redone like a week ago, so.
- [41:53]
C
Well, I think that is a beautiful point to end on. Kubernetes is perhaps not geriatric, but just, you know, millennial Botox stage.
- [42:03]
B
Botox.
- [42:05]
C
Botox. And so we are looking at adopting AI tools. We have a variety of ways that we're adopting AI tools now. And if you would like to join us, I hope you've learned something about the different ways that you can contribute and get involved in the community. Thank you, Kat and Natalie for joining me today.
- [42:23]
D
You're welcome.
- [42:24]
E
Thank you.
- [42:27]
C
I'm so excited that we got to do this interview with Katie, Kat and Natalie. As a Kubernetes maintainer myself, I am having interesting experiences with AI in the open source ecosystem, but I am a lead of a SIG that is about contributor experience. I lead the communication sub project. A lot of the things that I do, we do some things in GitHub. We also do a lot of things in other Tools and docs also they get a lot of AI contributions for docs, so it's not always code. But talking with other maintainers about their experiences, whether it's code or docs or whatever part of the project is just so interesting. So I love that angle and I think a lot of people think about running AI on Kubernetes, but how AI is affecting Kubernetes, the open source project is also really interesting.
- [43:10]
B
Yeah, I mean I didn't listen to the episode, but I am assuming that as any other open source project, they are probably struggling a little bit with the amount of AI slop they receive. Like as maintainers.
- [43:20]
C
Yeah, I know you hadn't had a chance yet, but I wanted to talk with you about this anyway because I just love it to so much.
- [43:25]
B
Well, the topic is very interesting.
- [43:26]
C
It's so good.
- [43:27]
B
Like the impact of AI on open source in general is felt across the industry, but Kubernetes specifically is very large projects, right?
- [43:33]
C
Yes. And interestingly, very recently I saw a comment from Linus Torvalds about AI usage in Linux and their opinion, his opinion at least kind of mimics what I'm seeing in the Kubernetes community is that we see AI as a tool for engineers to use and it can be a very useful one. And so we want to enable engineers contributing, anyone contributing to Kubernetes to be able to use this tool. But we do still have to think about guidelines around licensing. That was one of the major things we talked about in the episode, is that there's a contributor license agreement and the AI can't sign it. So we can't technically have people saying that something was co authored by an agent because the agent isn't allowed to be an author legally.
- [44:15]
B
Yeah, that's an interesting one.
- [44:16]
C
So we, we have things like that and also we're a global project, we have folks all around the world. So privacy laws vary and security laws vary. So adopting new AI tools cost. Yeah, we have to think about these things.
- [44:34]
B
I think I saw the post that you mentioned by Linus Torvalds and I just remembered like few months ago he also published something around the line that he wrote a tool, he wrote like an app in Python using AI. And the Internet wants absolutely bananas because they were like, see, even Linux Storvas is using AI.
- [44:51]
C
See that happening?
- [44:52]
B
And he had to come back and say like no, no, no, calm down, this is not actually I'm not using AI to write the Linux kernel. This is not C. This is just like an extra nice to have yeah, but, yeah, I agree. And I think a lot of people are hyping up AI. But like, if you think about it as a tool that can help you do stuff that you don't want to do, then it could be a very useful tool. Considering we sort out the details and
- [45:13]
C
the point in this AI usage policy that we're talking about and what I'm hearing from other open source maintainers and other projects sometimes too is the responsibility stays with the human.
- [45:23]
B
Yeah, of course.
- [45:24]
C
Yeah, you hear that all across the industry, right. That it's really important for the human to understand what the code is. Another thing we talk about in the episode and which I wanted to mention about AI slop contributions to projects.
- [45:38]
B
Yes.
- [45:39]
C
When I asked other maintainers about this, something they said is that they're not being so overwhelmed with new AI PRs, because they were already overwhelmed. So they're just as overwhelmed as they were before. But they are being frustrated and slowed down by AI created comments on PRs because they will just post pages and pages and it's unparsable, unreadable, and it can make things up and they just can't trust the comment to help them understand the code that's being proposed. So they really want to have conversations with humans in the comments. Even if AI was used to help write the code.
- [46:18]
B
Yeah. I can see how this can become a problem. I think it's the same thing happening in the security space. I was, when I was chatting with Michelle Shubrica to, to bring her to the podcast for another episode.
- [46:27]
C
Look forward to that.
- [46:28]
B
Yeah, we were talking about AI slope for security people. Like if you use AI to find vulnerabilities and report them, then at the end of the day there is still a human being that has to go and read through all this slop to identify is this actually real or is this just like a hypothetical thing that an AI hallucinated. Right. But we also see it, I think in our day work on gravity. A lot of times we just invent function names. It doesn't exist. Right.
- [46:51]
C
Yeah. And you can do AI as a judge to try to make sure that you're not, you know, your output, your final thing is not hallucinating and all of that. But once again, the AI is also fallible and the AI is not responsible is what it comes down to.
- [47:07]
B
I know a developer, I was talking to one of the founders of the Devox the conference and Stefan, he was telling me that he had three different subscriptions, three different models. So he will have one model generate code, another Model review the code that the first model generated and then another model reviewing the code. And then I was just asking him like, how much are you paying per month for all these subscriptions? Must be insane, right? Yeah, because it's like three different providers. It's like Google, Claude and whatever. Right. So if you are using the max subscription of all of them, there must be at least like thousand dollars a month just in subscription fees.
- [47:38]
C
That'd be intense.
- [47:39]
D
Yeah.
- [47:40]
B
And I guess also that for a project like Kubernetes, they have to think also about this in the sense of like cost. Right. Because if you build AI to the pipelines, how much tokens is this thing going to consume?
- [47:49]
C
Also an interesting thing that I didn't mention in this interview is, is that I've heard from other Kubernetes maintainers a few months ago. I don't know if it's still true right now, but when we were talking about an increase in AI contributions and stuff, we also saw an increase in CI CD pipeline attacks.
- [48:05]
B
Oh yeah, of course.
- [48:05]
C
Because people will attack our CICD pipeline where you create a PR in GitHub and we have automation bots that pick up that PR and run tests on them. So we have a flag that says okay to test and we have to have a reviewer say that the code is okay to run through our testing infrastructure. Because people will attack our CI CD pipeline to try to get the compute that we're running for our CICD and use it for their own purposes.
- [48:33]
B
Yeah, or just to financially hurt you. Right. By making you consume resources.
- [48:37]
C
Because boy, does the Kubernetes project use a lot of money for an open source project that doesn't directly bring in any money.
- [48:44]
B
I think it's like what, $3 million from Google, $3 million from Amazon, and I think Microsoft throws a couple of millions as well.
- [48:50]
C
It's a lot, I don't think. I feel like it was a hundred thousand, not million, but I could be wrong on that. I think we have blog posts about it. Definitely check it out though.
- [48:58]
B
I might be wrong, but I'm quite confident it's in the millions. It might be because you remember a couple of years ago when they had that issue, they were running out of credits because they were serving too much artifacts.
- [49:06]
C
Actually it happens every year, but it was a big deal that year.
- [49:09]
B
That year was a big deal. Yeah, I remember. It was like billions. It's millions of dollars they get in cloud credits to be able to run the entire test infrastructure and stuff.
- [49:17]
C
Yeah, I don't know if we've had someone on from SIG Architecture to talk about how much it costs to run Kubernetes Architecture, but we should think about that.
- [49:24]
B
Actually it would be interesting to bring somebody to talk about the CI thing. They have a CI tool that use
- [49:30]
C
it's technically within Mysig Prow.
- [49:33]
B
Yeah, correct.
- [49:33]
C
We have a few different tools but Prow is our main bottom proud bot.
- [49:38]
B
Yeah, it's interesting. This whole thing of like AI in open source is an interesting topic.
- [49:42]
C
We hope you enjoyed listening to the episode today. Let us know if you want more.
- [49:47]
B
Thank you for taking the time to do the episode.
- [49:48]
C
Thanks for being here. Abdel.
- [49:50]
B
Thank you.
- [49:50]
C
See you next time.
- [49:51]
B
See ya.
- [49:55]
A
That brings us to the end of another episode. If you enjoyed the show, please help us spread the word and tell a friend. If you have any feedback for us, you can find us on social media Kubernetes Pod or reach us by email@kubernetes podcastoogle.com youm can also check out the website@kubernetes podcast.com where you'll find transcripts, show notes and links. To subscribe, please consider rating us in your podcast player so we can help more people find and enjoy the show.
- [50:20]
C
Thanks for listening and we'll see you next time.
- [50:30]
B
Sa.