Kubernetes Podcast from Google: Episode Summary

Title: Working Group Serving, with Yuan Tang and Eduardo Arango
Hosts: Abdel Sghiouar, Kaslin Fields
Release Date: October 31, 2024

Introduction

In this episode of the Kubernetes Podcast from Google, hosts Abdel Sghiouar and Kaslin Fields engage in an insightful conversation with Yuan Tang and Eduardo Arango. The discussion centers around the newly formed Working Group Serving within the Kubernetes project, which focuses on optimizing serving mechanisms for AI and ML workloads. This summary encapsulates the key points, discussions, insights, and conclusions drawn during the episode.

News Highlights

Before delving into the main topic, Abdel and Kaslin share recent developments in the Kubernetes and cloud-native ecosystem:

1. Docker's Terraform Provider Launch
   [00:33] Kaslin announces that Docker has launched its official Terraform provider, enabling the management of Docker-hosted resources such as repositories, teams, and organization settings.

2. Open Collaboration Between Tritate and Bloomberg
   [00:47] Abdel highlights a collaboration aimed at integrating AI gateway features into the Envoy projects. This initiative focuses on building gateways capable of handling AI traffic, with initial features targeting usage limiting based on input and output tokens.

3. CNCF Laptop Drive at KubeCon
   [01:20] Kaslin informs listeners about the CNCF's laptop drive at KubeCon Cloud Native Con North America 2024, benefiting nonprofit organizations like Black Girls Code and Kids on Computers.

4. Upcoming Kubernetes Community Days
   [01:43] Abdel mentions four remaining Kubernetes Community Days events scheduled globally, encouraging listeners to participate and support local Kubernetes communities.

Introducing the Guests

Yuan Tang
Yuan Tang is a Principal Software Engineer at Red Hat, working on OpenShift AI. With a rich background in leading AI infrastructure and platform teams, Yuan holds leadership positions in open-source projects including Argo, Kubeflow, and the Kubernetes Working Group Serving. He is also an accomplished author and regular conference speaker.

Eduardo Arango
Eduardo Arango transitioned from environmental engineering to software engineering and has been instrumental in promoting containerized environments for high-performance computing (HPC) over the past eight years. As a core contributor to Apptainer under the Linux Foundation, Eduardo now works at NVIDIA on the Core Cloud Native Team, focusing on integrating specialized accelerators into Kubernetes workloads.

Understanding the Working Group Serving

Creation and Mission

The Working Group Serving was established following discussions at KubeCon Europe, where Yuan Tang and Clayton Cotton identified the need to address challenges in model serving within Kubernetes. The group's mission is to enhance serving workloads on Kubernetes, particularly for hardware-accelerated AI and ML inference tasks.

Yuan Tang:
"[03:53]…Generative AI has really introduced a lot of challenges and complexity in model serving systems… The mission is to advance the capabilities and efficiency of serving on Kubernetes to handle evolving requirements of generative AI and future workloads."

Goals of the Working Group Serving

Eduardo outlines three primary goals of the working group:

1. Enhancing Workloads Controllers
   [11:16] The group aims to provide recommendations and better patterns for improving Kubernetes workloads and controllers, focusing on performance enhancements in popular inference serving frameworks.

2. Orchestration and Scalability Solutions
   [11:16] Investigating key metrics for auto-scaling, such as GPU utilization and token-based metrics, to build more effective orchestration and load balancing solutions.

3. Optimizing Resource Sharing
   [11:16] Collaborating with the Working Group Device Management to communicate resource needs and prioritize new features like Dynamic Resource Allocation (DRA) for better resource sharing.

Key Challenges and Solutions

1. Multi-Host and Multi-Node Serving

As AI models grow in size, deploying them across multiple nodes becomes essential. Yuan explains the complexities involved in multi-host inference workloads, such as network topology preferences and load balancing across nodes.

Yuan Tang:
"[15:04]…auto scaling on device utilization, memory is really not sufficient for production workloads… it's challenging to configure HPA to autoscale model serving metrics."

2. Auto-Scaling Limitations

Traditional auto-scaling metrics, like memory utilization, fall short for AI workloads. The working group is exploring alternative metrics, including latency, tokens per second, and prompt sizes, to achieve more efficient scaling.

Eduardo Arango:
"[23:16]…the auto scaling work stream has been focusing on caching and metrics. If the model is not cached effectively, latency increases, degrading user experience."

3. Dynamic Resource Allocation (DRA)

DRA aims to address the limitations of defining multi-GPU and multi-node workloads in Kubernetes. Eduardo emphasizes the necessity of DRA for running large models that exceed the capacity of single nodes.

Eduardo Arango:
"[14:03]…defining multi-GPU multi-node workloads in Kubernetes is almost impossible without DRA. It's essential for models that require more GPUs than a single node can provide."

Technical Deep Dive

LLM Instance Gateway

A significant focus is placed on the LLM Instance Gateway, a sub-project designed to optimize load balancing for Large Language Models (LLMs). This gateway intelligently routes requests based on input size and model efficiency, ensuring low latency and optimal resource utilization.

Yuan Tang:
"[27:42]…the LLM Instance Gateway aims to safely multiplex use cases onto a shared pool of model servers for higher efficiency."

Model Mesh Integration

Yuan discusses the integration of Model Mesh, a mature model serving management layer that works seamlessly with existing model servers. It acts as a distributed LRU cache, enhancing performance for both traditional ML models and LLMs.

Yuan Tang:
"[29:02]…Model Mesh is not just for large models but also benefits traditional machine learning models by managing traffic and routing effectively."

Community and Collaboration

The Working Group Serving operates within the Kubernetes community, adhering to the CNCF Code of Conduct. It collaborates closely with ecosystem projects like KServe, Ray, and the Device Management Working Group to ensure holistic solutions for serving workloads.

Yuan Tang:
"[09:25]…We hope that all improvements made by the working group will also benefit other serving workloads like web services or stateful applications."

How to Get Involved

Yuan and Eduardo encourage community members to join the Working Group Serving by subscribing to the mailing list and participating in the Slack channel. Interested individuals can contribute by sharing use cases, proposing features, and collaborating on ongoing projects.

Yuan Tang:
"[29:18]…Make sure you join the mailing list and the Slack channel to stay updated and contribute to the discussions."

Conclusion

The episode provides a comprehensive overview of the Working Group Serving, highlighting its mission to enhance Kubernetes for AI and ML workloads. Through collaborative efforts and community involvement, the group seeks to address critical challenges like multi-host serving, auto-scaling, and dynamic resource allocation, paving the way for more efficient and scalable AI deployments on Kubernetes.

Kaslin Fields:
"[37:45]…We hope that you all enjoyed listening to the episode and learning about what the community is doing to support serving workloads in the distributed system that is Kubernetes."

Join the Conversation

For listeners interested in diving deeper or contributing to the Working Group Serving, detailed information and resources are available in the episode's Show Notes and the GitHub Repository. Engaging with the community through mailing lists and Slack channels is encouraged to stay abreast of the latest developments and contribute to shaping the future of AI serving on Kubernetes.

Connect with Hosts:

• Twitter: @KubernetesPod
• Email: kubernetespodcast@google.com
• Website: kubernetespodcast.com