Kubernetes Podcast from Google: Episode Summary

Title: 65k Nodes on GKE, with Maciej Rozacki and Wojciech Tyczyński
Hosts: Kaslyn Fields and Abdel Sigewar
Guests: Maciej Rozacki (Product Manager, GKE for AI Training) and Wojciech Tyczyński (Engineering Lead, GKE)
Release Date: November 13, 2024

Introduction

In Episode 65 of the Kubernetes Podcast from Google, hosts Kaslyn Fields and Abdel Sigewar delve deep into the remarkable expansion of Google Kubernetes Engine (GKE) to support clusters with up to 65,000 nodes, a significant leap from the previous 15,000-node limit. Joining them are Maciej Rozacki and Wojciech Tyczyński, who provide insights into the technical advancements, engineering challenges, and the broader implications of this monumental update in the Kubernetes ecosystem.

Background: Scaling Kubernetes for the AI Revolution

The era of Artificial Intelligence (AI) has exponentially increased the demand for colossal computational resources. Traditional Kubernetes clusters, suited for microservices and high-performance computing (HPC) workloads, grappled with scalability constraints as AI models grew in complexity and size. GKE’s new support for 65,000-node clusters is a direct response to these evolving needs, enabling seamless training and serving of AI models with unprecedented scale.

[03:08] Maciej Rozacki: “There is a clear demand for customers to start running at a much larger scale than before... to meet the needs of customers, to be able to both train and serve these models, we need to innovate both in the sizes of clusters and in the capabilities of hardware that they run with.”

GKE’s Leap to 65,000 Nodes: What It Means

GKE’s announcement signifies a fourfold increase in the maximum supported nodes per cluster, propelling it to an industry-leading position. This enhancement is meticulously engineered to support AI training at scales previously unattainable, accommodating models with up to 1 trillion parameters and paving the way for even larger models in the future.

[04:50] Wojciech Tyczyński: “We were able to offer cloud customers the ability to operate at 65,000 VM nodes Type of computing power just in a single cluster.”

Key Implications:

• Enhanced AI Capabilities: Facilitates training and serving of expansive AI models.
• Operational Flexibility: Allows mixed workloads, enabling both AI training and inference within a single cluster.
• Infrastructure Efficiency: Optimizes resource utilization, crucial given the scarcity of hardware and power resources.

Engineering Challenges and Solutions

Achieving support for 65,000-node clusters was no small feat. The journey involved overcoming numerous technical hurdles, primarily focused on enhancing Kubernetes' core architecture to handle such scale.

1. Control Plane Overhaul

One of the most significant changes was replacing the traditional etcd datastore with a Spanner-based storage solution. This shift was pivotal in making the Kubernetes control plane stateless, enhancing flexibility and scalability.

[11:19] Wojciech Tyczyński: “We were replacing etcd with our own GKE-specific storage. We call it Spanner-based storage because underneath it's using Spanner, which is Google's technology for database solutions.”

Benefits:

• Stateless Control Plane: Facilitates horizontal scaling of the control plane without performance degradation.
• Improved Reliability: Reduces the load on storage systems, enhancing overall system stability.

2. Data Plane Enhancements

Investments were made in the data plane to handle increased network traffic efficiently. This included optimizing connectivity and improving components like Cilium, a popular networking solution in Kubernetes.

[16:23] Wojciech Tyczyński: “We did a bunch of improvements across not just core Kubernetes but also in other projects... our engineers contributed back to upstream Cilium.”

3. API and Scheduler Improvements

To support AI workloads, several Kubernetes APIs were extended. Innovations such as dynamic resource allocation and advanced scheduling paradigms were introduced to manage the complex dependencies and resource requirements of AI tasks.

[12:52] Maciej Rozacki: “Dynamic resource allocation is a whole domain of how do you model this very advanced and sophisticated hardware... enabling these large AI platforms.”

4. Batch and Serving Workgroups

The establishment of specialized working groups like the Batch Working Group and the Serving Working Group ensured focused development on use-case-specific enhancements, facilitating better integration of AI and HPC workloads.

[10:00] Kaslyn Fields: “We have a bunch of scalability-related improvements going directly into Cilium... ensuring clusters of such scale actually work too.”

Contributions to Open Source Kubernetes

A cornerstone of GKE’s scalability advancements is the extensive contributions made to the open-source Kubernetes project. These enhancements not only powered GKE’s 65,000-node clusters but also benefit the broader Kubernetes community.

Key Contributions:

• Spanner-Based Storage Integration: Making control plane storage more scalable and flexible.

• Consistent List from Cache: Reduces API server load by serving list requests directly from the cache.

  [17:07] Wojciech Tyczyński: “Consistency list from cache allows us to serve the list request directly from API server cache without contacting etcd... reducing the load on the storage.”

• Advanced Scheduling Mechanisms: Incorporating AI-specific scheduling requirements to handle multi-host workloads and dynamic resource allocation.

[15:54] Maciej Rozacki: “Leader worker set to enable these more complicated deployments... balancing capacity sharing between jobs and your serving workloads.”

• Enhanced Networking with Cilium: Optimizing network traffic management to support large-scale clusters.

These contributions enhance Kubernetes’ core capabilities, enabling more users to leverage high-scale clusters irrespective of their specific use cases.

Benefits for Kubernetes Users

The expansion to 65,000-node clusters brings a myriad of benefits not just for AI-centric workloads but for the entire Kubernetes ecosystem.

1. Unified Workloads

Users can now train and serve AI models within the same cluster, enhancing operational efficiency and reducing the complexity of managing separate environments.

[22:00] Maciej Rozacki: “Unlike other systems that were built primarily with supercomputing in mind, Kubernetes was built both with supercomputing and these research workloads and the microservices... enabling customers to run both in one environment.”

2. Resource Flexibility

The ability to rapidly repurpose hardware allows users to adapt to fluctuating demands, crucial for AI research and deployment.

[09:15] Wojciech Tyczyński: “Being able to have that capacity without provisioning the accelerators... is an important factor for why users choose to repurpose existing capacity.”

3. Enhanced Reliability and Performance

Improvements such as serving list requests from cache and optimizing the control plane ensure that Kubernetes remains reliable and performant, even under high load.

[19:56] Wojciech Tyczyński: “None of these improvements just help for the size of the cluster, but they also make the system itself more reliable, reduce cliffs, and help with stability under high load.”

4. Open Source Benefits

All enhancements are contributed back to the Kubernetes open-source project, ensuring that even users operating smaller clusters reap the benefits of improved scalability, reliability, and performance.

[18:19] Kaslyn Fields: “The engineers contribute back to open source and everything is available in open source, allowing anyone to use these improvements.”

KubeCon 2024: Showcasing Innovations and Community Engagement

Released concurrently with the GKE announcement, KubeCon Cloud Native Con North America 2024 serves as a platform to showcase these advancements and foster community engagement.

Highlights from the Episode:

• AI Day Presentations: Featuring collaborations between Google engineers and partners like Apple, demonstrating sophisticated multitenant environments for researchers.

  [25:24] Maciej Rozacki: “Our engineers, together with our customers and partners from the community, will be presenting a couple of very interesting things.”

• Poster Sessions: Encouraging researchers to share their work and engage with the Kubernetes community.

  [26:23] Wojciech Tyczyński: “Just asking any of those people involved in the 65k nodes clusters will easily redirect you to someone you can speak to.”

• Maintainer Track Sessions: Offering deep dives into Kubernetes' core components and allowing attendees to interact directly with maintainers.

  [30:45] Kaslyn Fields: “At a maintainer track session, you know that you're talking directly to the engineers who are influencing those areas of the Kubernetes project.”

Community Recommendations:

• Engage with Maintainer Tracks: These sessions provide invaluable insights into the future direction of Kubernetes and offer opportunities to influence its development.
• Explore Open Source Contributions: Attendees are encouraged to explore the numerous features and improvements that enable large-scale Kubernetes deployments.

[27:57] Kaslyn Fields: “It's very important to celebrate that work and for people to know about how awesome it is, which we will be doing at KubeCon.”

Key Takeaways and Future Outlook

The expansion of GKE to support 65,000-node clusters is a testament to Kubernetes’ evolving capabilities in the face of burgeoning AI demands. This leap not only reinforces GKE’s position as a leading managed Kubernetes service but also underscores the collaborative spirit of the open-source community in driving technological innovation.

Scalability as a Multi-Dimensional Challenge

The podcast emphasizes that scalability in Kubernetes is a multi-faceted problem, involving various components and layers that must work in harmony. Understanding these interdependencies is crucial for effectively scaling Kubernetes clusters.

[43:35] Kaslyn Fields: “Keeping in mind that scalability is a multi-dimensional problem... all of those are valid.”

Community and Collaboration

The strides made in scaling Kubernetes are a direct result of sustained community collaboration and open-source contributions. The hosts encourage listeners to engage with the community, participate in events like KubeCon, and contribute to the ongoing evolution of Kubernetes.

[34:50] Maciej Rozacki: “We will be posting information on our cloud blog... reach out directly to us or to your account team.”

Future Innovations

As AI models continue to grow, further innovations in Kubernetes’ architecture and capabilities will be necessary. The groundwork laid by GKE’s 65,000-node support sets the stage for future advancements, ensuring Kubernetes remains at the forefront of cloud-native computing.

Closing Thoughts

Episode 65 of the Kubernetes Podcast from Google offers a comprehensive look into the monumental scaling achieved by GKE and the collaborative efforts that made it possible. Maciej Rozacki and Wojciech Tyczyński provide invaluable insights into the engineering feats, open-source contributions, and the broader implications for the Kubernetes ecosystem. As Kubernetes continues to evolve, its resilience and adaptability shine through, solidifying its role as a cornerstone of cloud-native infrastructure.

[36:23] Abdel Sigewar: “None of this would be possible without all the years of contribution and improvements into Kubernetes open source.”

Listeners are encouraged to attend KubeCon, explore the latest features, engage with maintainers, and contribute to the ongoing success of Kubernetes.

Resources:

• Follow Hosts on Twitter: @KubernetesPod
• Email: kubernetespodcast@google.com
• Website: kubernetespodcast.com
• OpenTelemetry Release Notes: Referenced in show notes
• Gitpod Blog Post on Kubernetes: Referenced in show notes