Podcast Summary: Software Engineering Daily – ByteDance’s Container Networking Stack with Chen Tang

Episode Overview

Title: ByteDance’s Container Networking Stack with Chen Tang
Host/Author: Software Engineering Daily
Release Date: July 1, 2025

In this episode of Software Engineering Daily, host Kevin Ball engages in an in-depth discussion with Chen Tang, a software engineer at ByteDance. They delve into ByteDance's innovative use of Extended Berkeley Packet Filter (eBPF) technology to overhaul the company's container networking stack. The conversation explores the challenges of operating at ByteDance's massive scale, the intricacies of eBPF, and the future of kernel vs. user-space networking solutions.

Introduction to ByteDance and Chen Tang’s Role

[00:00] Chen Tang:
Chen introduces ByteDance as a global technology leader behind platforms like TikTok, highlighting the immense scale of over a million servers running containerized applications. He emphasizes the necessity of a robust networking solution to maintain performance and stability across data centers.

[01:52] Chen Tang:
“We use a lot of different technologies, we using kernel technology and hardware technologies, but in the kernel part we use eBPF... you don’t need to be afraid that your code might jeopardize the entire system.”

Understanding eBPF and Its Advantages

[02:54] Kevin Ball:
Kevin compares eBPF to sandboxed environments like JavaScript in browsers, making it more accessible to developers by simplifying kernel interactions.

[03:20] Chen Tang:
Chen agrees, explaining eBPF as a virtual machine within the kernel that safely executes custom programs without the need for kernel modules.

[05:20] Kevin Ball:
He clarifies that eBPF provides stable APIs or hooks into the kernel, complemented by static analysis to ensure safety before execution.

Transitioning to eBPF in Container Networking

[06:08] Chen Tang:
Chen discusses the challenges of container networking, especially with unique network namespaces and the need to efficiently route packets between containers and the host. He explains how eBPF enables a decentralized networking solution by capturing and analyzing packets to direct them appropriately.

[10:45] Chen Tang:
“As the functions exposed initially were not enough, we faced performance issues and complexity in writing eBPF programs. Over five years, the community has optimized eBPF to make it mature for container networking.”

Scaling Cloud Native Technologies at ByteDance

[19:49] Kevin Ball:
Kevin highlights the unique scalability challenges ByteDance faces compared to companies like Amazon or Meta, which use Kubernetes primarily as a service provider, handling smaller clusters up to 1,000 machines.

[22:26] Chen Tang:
Chen explains that ByteDance manages over a million servers, necessitating custom solutions to overcome Kubernetes' scalability limitations. For instance, they developed their own service discovery framework to manage the massive scale efficiently.

[24:50] Kevin Ball:
He underscores how cloud native abstractions can become bottlenecks at large scales, prompting the need for tailored optimizations.

Replacing Traditional Networking Components with eBPF

[27:29] Kevin Ball:
Kevin asks about replacing virtual switches and iptables with eBPF.

[27:42] Chen Tang:
Chen details that iptables, though integral to kernel networking, operate inefficiently with extensive rule chains. eBPF offers a more streamlined approach by allowing single, custom programs for packet handling, enhancing performance and manageability.

[28:45] Chen Tang:
He differentiates between using eBPF for container networking and virtual switches for virtual machines, explaining that eBPF is optimal for lightweight, isolated environments like containers, whereas virtual switches are necessary for the full isolation required by virtual machines.

Notable Quote:

Chen Tang:
“eBPF is a single program. You write your own program and one program will be enough. So that's why people replace iptables and choose to use eBPF.”

Integrating eBPF with Hardware Offloading and RDMA

[33:25] Chen Tang:
Chen discusses the ongoing efforts to minimize kernel stack costs by integrating eBPF with hardware offloading technologies like SmartNICs. This combination aims to translate eBPF programs into hardware rules, thereby enhancing packet processing efficiency.

[37:57] Chen Tang:
He explains the current method of using an agent to manage rule translation and injection into hardware, acknowledging its complexity but recognizing it as a necessary step until more seamless integration solutions emerge.

[42:10] Chen Tang:
Chen elaborates on leveraging RDMA (Remote Direct Memory Access) to bypass the kernel entirely, using eBPF to direct packets directly to containers, thus further reducing latency and overhead.

Notable Quote:

Chen Tang:
“We have a separation of the slow path and fast path. The fast path is hardware offloading. And once the packet misses the rules and the packet will go back to the kernel again and eBPF program will process the packet.”

Future Directions and the Kernel vs. Bypass Technology Debate

[45:55] Chen Tang:
Chen raises a thought-provoking point about the future coexistence of kernel-based and bypass technologies, pondering whether they will complement each other or become mutually exclusive.

[47:42] Chen Tang:
He acknowledges the potential of eBPF as a middle ground, offering dynamic, safe kernel modifications without the overhead traditionally associated with kernel programming.

Notable Quote:

Chen Tang:
“We are facing a choice and I think it's a very interesting topic and actually I don't have the answer yet because we are still wondering if they can coexist in the future or they have to be enemies.”

Conclusion

The conversation between Kevin Ball and Chen Tang offers a comprehensive look into how ByteDance leverages eBPF to address the complexities of networking at an unprecedented scale. From replacing traditional tools like iptables and virtual switches to integrating advanced hardware offloading and RDMA technologies, ByteDance exemplifies the cutting-edge application of eBPF in cloud-native environments. The discussion also touches on broader industry questions about the future interplay between kernel-based and user-space networking solutions, highlighting the ongoing evolution of software engineering practices in managing large-scale infrastructures.

Key Takeaways:

1. eBPF as a Catalyst for Networking Efficiency: ByteDance's implementation of eBPF allows for dynamic, safe, and efficient packet processing within the kernel, replacing older, less efficient tools.

2. Scalability Challenges: Managing over a million servers necessitates custom solutions beyond traditional Kubernetes deployments, particularly in service discovery and network management.

3. Integration with Hardware Technologies: Combining eBPF with hardware offloading and RDMA presents opportunities to further reduce latency and overhead, though it introduces complexity in rule management.

4. Future of Kernel vs. Bypass Technologies: The industry faces pivotal decisions on whether to continue enhancing kernel-based solutions like eBPF or to shift towards bypass technologies, with potential for both coexistence and competition.

Relevant Quotes:

• Chen Tang: “You just inject your code inside the running system and you get everything you need. And once it's done and you just cancel it, you can remove the program and the system will become back to normal.” [14:38]

• Chen Tang: “We have iptables that is old technologies being first used when cloud native actually at the very beginning, when container networking becomes a problem to be solved, people use iptables at first, but iptables, it's slow.” [26:56]

• Chen Tang: “Because we use ourselves. So the difference is since we have the kubernetes running inside our clusters and I think the biggest difference is the problem of scalability.” [20:15]

This episode provides valuable insights for software engineers and technologists interested in large-scale cloud-native networking solutions, the practical applications of eBPF, and the future trajectory of kernel and user-space networking technologies.