Replacing kube-proxy with eBPF: A Production Case Study on Latency Reduction and the Monitoring Pitfalls

The Kubernetes networking stack has long been defined by a single, ubiquitous component: kube-proxy. For years, it has served as the indispensable glue translating Service definitions into iptables or IPVS rules, ensuring traffic reaches the correct pods. However, as clusters scale into the thousands of nodes and tens of thousands of services, the traditional userspace-to-kernelspace context switches and linear rule lookups of kube-proxy become a bottleneck, introducing unpredictable latency spikes and consuming valuable CPU cycles on every node.

The migration to eBPF-based networking, primarily driven by projects like Cilium, offers a radical departure from this paradigm. By replacing kube-proxy with eBPF programs that run directly in the kernel, we can achieve sub-microsecond packet processing, deterministic latency, and significantly reduced CPU overhead. But this shift is not without its costs. The transparency of the traditional stack is replaced by the complexity of eBPF verification, and monitoring becomes a double-edged sword: you gain visibility into packet paths but lose the familiar metrics provided by standard CNI plugins.

This article presents a production case study of migrating a large-scale Kubernetes cluster (500+ nodes, 10,000+ services) from kube-proxy (IPVS mode) to Cilium with eBPF. We will analyze the quantitative improvements in latency and resource usage, but more importantly, we will dissect the "monitoring pitfalls"—the subtle blind spots that caught us off guard during the rollout and how we engineered solutions to observe our new stack effectively.

The Architecture: From iptables to eBPF

To understand the benefits, we must first contrast the underlying mechanisms. In a standard Kubernetes setup, kube-proxy watches the API Server for Service and Endpoint changes. When it detects a change, it updates the kernel's netfilter subsystem (iptables or IPVS).

The Limitations of IPVS

Even in IPVS mode, which is faster than iptables, there are inherent inefficiencies:

1. Context Switching: Every packet traversing the network stack must jump from user space (where kube-proxy logic resides) to kernel space (where iptables rules are applied). While IPVS minimizes this, it still involves multiple lookups.
2. Rule Explosion: As the number of services and endpoints grows, the number of iptables rules grows linearly or worse. On a 10,000-endpoint cluster, this can lead to significant CPU usage during rule synchronization events.
3. Lack of Deep Visibility: iptables operates at Layer 3/4. It cannot easily inspect Layer 7 context without heavy overhead (like conntrack), making advanced traffic management (like mTLS or fine-grained policy) difficult to implement efficiently.

The eBPF Advantage

eBPF (extended Berkeley Packet Filter) allows us to load custom programs into the kernel safely. Cilium uses eBPF to replace the entire kube-proxy functionality. Instead of updating iptables rules, Cilium agents install eBPF programs at various hooks in the kernel's networking stack (e.g., tc hooks, XDP, or netfilter hooks).

The key advantages are:

• Deterministic Performance: eBPF programs are JIT-compiled and run in a sandboxed VM within the kernel. Lookups are often hash-based or tree-based, providing O(1) or O(log n) performance regardless of cluster size.
• No Userspace-Kernelspace Boundaries: For many operations, the packet processing stays entirely within the kernel, eliminating context switches.
• Rich Context Awareness: eBPF programs can access deep packet context, enabling Layer 7 visibility and policy enforcement without performance penalties.

Case Study: Production Migration Results

Our cluster consists of 500+ nodes running mixed workloads, including high-frequency trading microservices and batch processing jobs. We measured the impact of the migration over a two-week period, comparing the final week of kube-proxy (IPVS) with the first two weeks of Cilium (eBPF).

1. Latency Reduction

The most immediate and tangible benefit was the reduction in P99 and P99.9 latency for inter-service communication.

The reduction in tail latency is particularly critical for our trading services, where consistency matters more than average performance. The eBPF-based load balancing ensures that packet processing times are highly predictable, eliminating the spikes caused by iptables rule synchronization and hash table collisions.

2. CPU and Memory Overhead

kube-proxy is a stateful userspace process that must re-read the entire Kubernetes API state periodically. In our cluster, this resulted in a baseline CPU usage of ~5-10% on control-plane nodes and ~2-5% on worker nodes during sync events.

With Cilium, the kube-agent is still present but the heavy lifting is done by eBPF programs. The result was a dramatic drop in CPU usage on worker nodes.

The reduction in memory footprint is due to the elimination of the userspace process overhead and the efficient data structures used by eBPF maps.

3. Scalability

We stress-tested the cluster by adding 5,000 additional services and 50,000 endpoints. With kube-proxy, the API Server latency increased significantly due to the volume of updates, and kube-proxy pods struggled to keep up with the rule updates, leading to temporary service disruptions.

With Cilium, the eBPF maps scaled linearly, and the agent remained stable. The cluster handled the increased load without any noticeable degradation in performance.

The Monitoring Pitfalls: What Went Wrong

While the performance metrics were stellar, the migration exposed significant gaps in our observability stack. The traditional monitoring tools we relied on were ill-equipped to handle the nuances of eBPF-based networking. Here are the critical pitfalls we encountered and how we resolved them.

Pitfall 1: The "Black Box" of eBPF Maps

In the kube-proxy world, monitoring was straightforward: check the CPU usage of the kube-proxy pod and the rules in iptables. With Cilium, the actual packet processing happens in the kernel via eBPF maps. These maps are not easily inspectable by standard tools.

The Problem: We noticed occasional packet drops that were not reflected in the standard Kubernetes metrics. The drops were happening in the eBPF program itself, but our Prometheus exporters were not capturing them.

The Solution: We had to enable and scrape Cilium-specific metrics. Cilium exposes metrics via its agent, which can be scraped by Prometheus. We configured the cilium-agent to expose metrics on /metrics and created custom Prometheus rules to alert on cilium_drop_total and cilium_forward_total.

# prometheus_rules.yaml
groups:
- name: cilium-alerts
  rules:
  - alert: HighCiliumDropRate
    expr: rate(cilium_drop_total[5m]) > 0.1
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High packet drop rate in Cilium"
      description: "Cilium is dropping more than 0.1 packets per second."

We also had to install the cilium-monitor tool in our troubleshooting toolkit, which provides real-time visibility into eBPF events, including drops, retries, and policy decisions.

Pitfall 2: Loss of Standard CNI Metrics

Many of our existing dashboards were built around the metrics provided by the CNI plugin (Calico or Flannel). These metrics included packet counts, byte counts, and error rates at the interface level. When we switched to Cilium, these metrics disappeared because Cilium does not use standard Linux interfaces in the same way.

The Problem: Our Grafana dashboards showed zero traffic for several hours after the migration, leading to false alarms.

The Solution: We had to rebuild our dashboards using Cilium's native metrics. Cilium provides metrics for kube-proxy replacement, such as cilium_proxy_total, cilium_forward_total, and cilium_endpoint_policy_enforcement_total. We also had to rely on node_exporter for basic network interface metrics, but these are less granular than CNI-specific metrics.

Pitfall 3: Complexity in Troubleshooting Network Policies

Network policies in Kubernetes are a common source of confusion. With kube-proxy, policies were often enforced at the node level via iptables rules. With Cilium, policies are enforced via eBPF programs, which are more flexible but also more complex.

The Problem: A developer reported that a specific service could not reach another service, despite the network policy appearing to allow it. The standard kubectl get networkpolicy output did not reveal the issue.

The Solution: We had to adopt Cilium's cilium policy trace command, which provides a detailed trace of how a packet is processed through the eBPF programs and network policies. This command revealed that the policy was being enforced at the endpoint level, not the node level, and that the source IP was not matching the policy selector.

# Trace the policy for a specific packet
cilium policy trace --source 10.0.0.1 --destination 10.0.0.2 --dport 80

This command provided a step-by-step breakdown of the packet's journey, including which eBPF programs were invoked and which rules were matched or denied.

Pitfall 4: Identity-Based vs. IP-Based Monitoring

Cilium uses Kubernetes identities (labels) to enforce policies, rather than IP addresses. This is a powerful feature but makes monitoring more abstract.

The Problem: Our monitoring tools were configured to alert on IP-based anomalies. When Cilium moved traffic between pods with the same IP but different identities, our alerts fired incorrectly.

The Solution: We had to update our monitoring infrastructure to understand Kubernetes identities. This involved mapping pod IPs to their corresponding labels and identities in our monitoring stack. We used Cilium's cilium endpoint list command to get the current mapping and integrated it into our alerting logic.

Implementation Strategy: How We Migrated

Migrating a production cluster is a high-risk operation. We adopted a phased approach to minimize disruption.

Phase 1: Parallel Run

We installed Cilium in parallel with the existing CNI plugin. We did not remove kube-proxy yet. This allowed us to observe the performance of Cilium in a non-disruptive manner.

Phase 2: Gradual Rollout

We migrated nodes one by one to use Cilium. We started with a small subset of nodes (5%) and monitored the performance and stability. We gradually increased the percentage until all nodes were running Cilium.

Phase 3: Disable kube-proxy

Once we were confident in Cilium's performance, we disabled kube-proxy on all nodes. This was the critical step that enabled the latency reductions and CPU savings.

# Disable kube-proxy on a node
cat <<EOF | kubectl apply -f -
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "disabled"
EOF

Phase 4: Optimization

We tuned Cilium's configuration to optimize for our specific workload. This included adjusting the eBPF load balancing algorithm, tuning the eBPF map sizes, and configuring network policies for maximum efficiency.

Production Best Practices for eBPF Networking

Based on our experience, here are the best practices we recommend for teams considering a migration to eBPF-based networking.

1. Invest in Observability Early: Do not assume that standard Kubernetes metrics will work. Plan for Cilium-specific metrics and tools like cilium-monitor and cilium policy trace from day one.
2. Test in Staging First: The behavior of eBPF programs can be subtle. Thoroughly test your network policies and traffic flows in a staging environment that mirrors production.
3. Monitor eBPF Map Sizes: eBPF maps have fixed sizes. Monitor the usage of these maps to ensure they do not fill up, which can lead to packet drops.
4. Understand the Policy Enforcement Model: Cilium uses identity-based policies. Ensure your team understands how identities are assigned and how policies are enforced at the endpoint level.
5. Plan for Rollbacks: Have a clear rollback plan in case the migration fails. This includes having the existing CNI plugin and kube-proxy configurations readily available.

Conclusion

Replacing kube-proxy with eBPF is not just a performance upgrade; it is a fundamental shift in how Kubernetes networking operates. The benefits in latency, CPU efficiency, and scalability are significant, especially for large-scale clusters. However, these benefits come with increased complexity in observability and troubleshooting.

The key to a successful migration is not just the technical implementation but also the preparation of your monitoring and troubleshooting tools. By addressing the monitoring pitfalls early and adopting a phased migration strategy, you can unlock the full potential of eBPF-based networking without compromising the stability of your production environment.

For more insights on Kubernetes networking and eBPF, check out Tamiz's Insights, where we regularly publish deep dives into the latest technologies and best practices for software engineers and systems architects.