Offensive Security Researcher & Quantum Cryptography Analyst

How to Configure Network Plugins in Kubernetes

Abdul Wahab Junaid
How to Configure Network Plugins in Kubernetes

Kubernetes has an odd property that surprises people coming from traditional infrastructure: it defines a networking model but ships with no default implementation of it. Without a CNI (Container Network Interface) plugin installed, nodes never leave the NotReady state and Pods can’t get IP addresses at all. This guide covers what CNI actually does, how to choose between the major plugins, and how to configure one properly.

The Kubernetes Networking Model

Kubernetes requires, by specification, that:

CNI plugins are what actually implement these guarantees at the networking layer — Kubernetes itself just calls the CNI plugin’s binary at Pod creation/deletion time and expects it to hand back a working network namespace.

Major CNI Plugin Options

Installing Calico

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml

Configure the pod CIDR to match what was set in kubeadm init --pod-network-cidr:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - blockSize: 26
        cidr: 192.168.0.0/16
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()

kubectl apply -f calico-installation.yaml

Verify nodes transition to Ready:

kubectl get nodes -w

NAME    STATUS   ROLES           AGE
cp1     Ready    control-plane   2m
node1   Ready    <none>          90s

Installing Cilium

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=lb.example.com \
  --set k8sServicePort=6443

Cilium’s kubeProxyReplacement=true mode uses eBPF to handle Service routing directly in the kernel, which can materially outperform iptables-based kube-proxy at scale — a common reason teams migrate to Cilium on larger clusters.

Check status:

cilium status

    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Envoy DaemonSet: OK
 \__/¯¯\__/    Hubble Relay:   OK
    \__/       ClusterMesh:    disabled

NetworkPolicy: The Payoff of a Real CNI

Flannel alone can’t enforce this — it requires Calico, Cilium, or another policy-capable plugin:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Apply and verify enforcement:

kubectl apply -f network-policies.yaml

# From a pod without the frontend label — should fail
kubectl run test --image=busybox --rm -it -- wget -qO- backend:8080 --timeout=3

Cilium-Specific Policy (L7-Aware)

Cilium extends NetworkPolicy with CiliumNetworkPolicy, capable of application-layer (HTTP method/path) rules:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-l7-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/.*"

This is a genuinely different capability tier than plain NetworkPolicy — restricting not just which Pods can talk, but which HTTP methods and paths are allowed between them.

Choosing Between CNI Plugins

For most new production clusters today, the practical choice comes down to Calico (mature, well-documented NetworkPolicy, BGP flexibility) versus Cilium (eBPF performance, Hubble observability, L7 policy). Flannel remains reasonable for learning environments or clusters with no NetworkPolicy requirement at all, but it’s rarely the right default for production.

RBAC Considerations

CNI DaemonSets typically run with elevated privileges (host networking, NET_ADMIN capability) since they configure the host’s network stack directly:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: calico-node
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["crd.projectcalico.org"]
    resources: ["*"]
    verbs: ["*"]

This is one of the few legitimately privileged workloads in a cluster — it’s also why CNI DaemonSets usually run in a namespace explicitly exempted from the restricted Pod Security Standard.

Monitoring Network Health

# Calico
calicoctl node status

# Cilium
cilium status --verbose
hubble observe --namespace production

Hubble in particular is worth calling out — it gives live, filterable flow visibility (which Pod talked to which, on what port, allowed or denied) that’s genuinely useful during both debugging and security review.

Troubleshooting Common Networking Failures

# Nodes stuck NotReady — usually means no CNI is installed yet
kubectl describe node node1 | grep -A 5 Conditions

# Pod stuck ContainerCreating — often a CNI failure
kubectl describe pod <pod-name> -n production

# Check CNI plugin logs
kubectl -n kube-system logs -l k8s-app=calico-node

IP Address Management (IPAM) Considerations

Every CNI plugin has to solve IP allocation somehow, and the approach affects both scale limits and cross-cloud portability. Calico, for instance, allocates IP blocks per node from the configured pool, and the block size determines how many Pods a single node can host before needing another block:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - cidr: 192.168.0.0/16
        blockSize: 26

A /26 block gives 64 addresses per allocation — comfortable for most nodes, but worth increasing (to /24, for instance) on clusters running unusually high Pod density per node. Running out of IP blocks manifests as Pods stuck in ContainerCreating with IPAM errors in the CNI plugin’s logs, which is a good first thing to check when Pod creation starts failing cluster-wide on an otherwise healthy-looking cluster.

MTU Mismatches: A Subtle, Recurring Problem

Overlay networking (VXLAN, IPIP) adds encapsulation overhead, which reduces the effective MTU available to Pod traffic below the underlying network’s MTU. When the CNI plugin’s configured MTU doesn’t account for this, the symptom is bizarre and hard to immediately connect to networking: large packets silently drop or fragment incorrectly, often manifesting as slow or hanging connections for larger payloads while small requests work fine.

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    mtu: 1440

A common baseline: standard Ethernet MTU is 1500; VXLAN encapsulation overhead is typically 50 bytes, so an overlay MTU of 1450 or lower avoids fragmentation. Cloud environments with their own overlay (like AWS VPC in some configurations) may need an even lower value. If application traffic looks fine for small API calls but times out or stalls on larger payloads (file uploads, big JSON responses), an MTU mismatch is worth ruling out early rather than assuming it’s an application bug.

Dual-Stack (IPv4/IPv6) Networking

For clusters needing IPv6 support alongside IPv4, most modern CNI plugins support dual-stack configuration, though it needs to be enabled consistently across the API server, kubelet, and CNI plugin simultaneously:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: VXLANCrossSubnet
      - cidr: fd00:10:244::/56
        encapsulation: None

kubeadm init --pod-network-cidr=192.168.0.0/16,fd00:10:244::/56 \
  --service-cidr=10.96.0.0/12,fd00:10:96::/112

Overlay Networking vs. Native Routing: The Underlying Trade-off

It’s worth understanding the actual architectural choice most CNI plugins are making, since it explains many of their performance and operational differences. Overlay networks (VXLAN, IPIP) encapsulate Pod traffic inside packets routable over the existing physical network, meaning the underlying infrastructure needs no awareness of Pod IPs at all — this makes overlay networking portable across almost any environment, at the cost of encapsulation overhead and the MTU considerations covered above. Native/BGP-based routing (Calico’s default mode outside of overlay, or cloud-native CNIs like AWS VPC CNI) instead advertises Pod IPs as real, directly routable addresses on the underlying network, avoiding encapsulation overhead entirely but requiring the physical network to actually support it — BGP peering with physical routers, or direct VPC IP allocation on a cloud platform.

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
      - cidr: 192.168.0.0/16
        encapsulation: None
    bgp: Enabled

The practical guidance: overlay mode is the safer default for most environments (works anywhere, no special network cooperation required), while native routing is worth the extra setup complexity specifically on clusters where Pod-to-Pod network throughput and latency are measurably load-bearing for the workloads running on them — high-frequency trading systems, latency-sensitive real-time processing, and similar cases where every microsecond of encapsulation overhead is actually noticed.

Switching CNI Plugins on a Running Cluster

A detail worth stating plainly since it surprises people: there is no supported, safe way to switch CNI plugins on a live cluster without disruption. Pod networking is established at Pod creation time by whichever CNI plugin is active, and changing plugins generally requires recreating every Pod in the cluster (a rolling node-by-node cordon-drain-replace, or a full blue-green cluster migration) rather than any kind of in-place transition. This is one of the more consequential early decisions in a cluster’s life — worth genuinely evaluating Calico vs. Cilium vs. a cloud-native option before the initial cluster build, rather than treating it as something easily revisited later.

Common Mistakes

Summary

CNI plugins are the unglamorous but mandatory foundation of a working Kubernetes cluster — nothing schedules correctly, and no NetworkPolicy can be enforced, without one properly installed. Calico and Cilium cover the vast majority of real production needs today, with the choice mostly coming down to whether eBPF performance and L7-aware policy (Cilium) matter more than maturity and BGP flexibility (Calico) for a given environment.

References

Exit mobile version