Offensive Security Researcher & Quantum Cryptography Analyst

How to Set Up a Highly Available Kubernetes Cluster

Abdul Wahab Junaid
How to Set Up a Highly Available Kubernetes Cluster

A single control-plane node is fine for a home lab. It is not fine for anything that matters, because the moment that one node goes down, you lose the ability to schedule, scale, or even query the state of your cluster — even if every application Pod keeps running untouched on the data plane. Building a highly available cluster means removing every single point of failure from the control plane, not just the workloads running on top of it.

What “Highly Available” Actually Means Here

There are two separate failure domains to address:

  1. Control plane HA — multiple API servers, multiple etcd members, no single node whose failure breaks cluster management.
  2. Workload HA — multiple replicas of applications, spread across failure domains, surviving node loss (covered by PodAntiAffinity, topology spread, and PodDisruptionBudgets elsewhere).

This article focuses primarily on the first, since it’s the part people most often get wrong or skip entirely.

Control Plane Architecture

A production HA control plane needs, at minimum:

           ┌────────────────────┐
           │   Load Balancer    │
           │  (kube-apiserver)  │
           └─────────┬──────────┘
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
   ┌─────────┐   ┌─────────┐   ┌─────────┐
   │ CP Node1│   │ CP Node2│   │ CP Node3│
   │apiserver│   │apiserver│   │apiserver│
   │  etcd   │   │  etcd   │   │  etcd   │
   │scheduler│   │scheduler│   │scheduler│
   │ ctrl-mgr│   │ ctrl-mgr│   │ ctrl-mgr│
   └─────────┘   └─────────┘   └─────────┘

Only one kube-scheduler and one kube-controller-manager are ever active at a time — they use leader election, so running 3 replicas of each is about failover speed, not concurrent operation.

Bootstrapping with kubeadm

Set up a load balancer first (HAProxy example):

# /etc/haproxy/haproxy.cfg
frontend kubernetes-frontend
  bind *:6443
  mode tcp
  default_backend kubernetes-backend

backend kubernetes-backend
  mode tcp
  balance roundrobin
  server cp1 10.0.1.11:6443 check
  server cp2 10.0.1.12:6443 check
  server cp3 10.0.1.13:6443 check

Initialize the first control-plane node, pointing at the load balancer’s address:

kubeadm init \
  --control-plane-endpoint "lb.example.com:6443" \
  --upload-certs \
  --pod-network-cidr=192.168.0.0/16

Output includes two join commands — save both:

You can now join any number of control-plane nodes by running:
  kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:1234... \
    --control-plane --certificate-key 5678...

You can now join any number of worker nodes by running:
  kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:1234...

Join the second and third control-plane nodes:

kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
  --discovery-token-ca-cert-hash sha256:1234... \
  --control-plane --certificate-key 5678...

Join worker nodes with the second command (no --control-plane flag).

Verifying the Control Plane

kubectl get nodes

Output:

NAME    STATUS   ROLES           AGE   VERSION
cp1     Ready    control-plane   10m   v1.30.2
cp2     Ready    control-plane   8m    v1.30.2
cp3     Ready    control-plane   7m    v1.30.2
node1   Ready    <none>          5m    v1.30.2
node2   Ready    <none>          5m    v1.30.2

Check etcd cluster health directly:

kubectl -n kube-system exec etcd-cp1 -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health --cluster

Output:

https://10.0.1.11:2379 is healthy: successfully committed proposal
https://10.0.1.12:2379 is healthy: successfully committed proposal
https://10.0.1.13:2379 is healthy: successfully committed proposal

Networking (CNI)

A CNI plugin has to be installed before nodes report Ready. Calico is a common production choice for its NetworkPolicy support:

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

RBAC for Cluster Administration

Once the cluster is up, scope access properly rather than distributing the admin kubeconfig broadly:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: platform-team-admin
subjects:
  - kind: Group
    name: platform-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Worker Node HA Considerations

Beyond the control plane, worker-side HA involves spreading nodes across availability zones and using cluster autoscaling to replace lost capacity automatically:

apiVersion: v1
kind: Node
metadata:
  labels:
    topology.kubernetes.io/zone: us-east-1a

Application-level HA (replica count, PodAntiAffinity, PodDisruptionBudgets) then layers on top of this zone-aware node topology.

Monitoring the Control Plane

kubectl get componentstatuses
kubectl -n kube-system get pods -o wide

For ongoing visibility, scrape control-plane metrics into the Grafana/Prometheus stack:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-apiserver
  namespace: monitoring
spec:
  selector:
    matchLabels:
      component: apiserver
  endpoints:
    - port: https
      scheme: https
      tlsConfig:
        insecureSkipVerify: true

Backup and Disaster Recovery

etcd is the single source of truth for cluster state — losing it without a backup means losing the cluster’s entire configuration, not just workloads:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Automate this with a CronJob or an external backup tool, and periodically test restoring it — an untested backup is not a real backup.

Testing HA Before You Need It

An HA control plane that has never actually been tested against a real node failure is a hypothesis, not a guarantee. Deliberately testing failure scenarios in a staging environment (never production, for obvious reasons) builds real confidence:

# Simulate losing one control-plane node
kubectl drain cp2 --ignore-daemonsets --delete-emptydir-data
sudo systemctl stop kubelet kube-apiserver etcd

# Confirm the cluster is still fully functional
kubectl get nodes
kubectl create deployment test --image=nginx
kubectl scale deployment test --replicas=5

If everything above still works with cp2 down, the load balancer and etcd quorum are doing their job correctly. Bring the node back and confirm it rejoins cleanly:

sudo systemctl start etcd kube-apiserver kubelet
kubectl get nodes
kubectl -n kube-system exec etcd-cp1 -- etcdctl endpoint health --cluster

Running this drill periodically — not just once at initial setup — catches configuration drift before an actual outage does. A control plane that was HA six months ago after a manual node replacement or a certificate rotation isn’t guaranteed to still be HA today without a periodic check.

What Happens When etcd Loses Quorum

Understanding the actual failure mode matters as much as preventing it. With 3 etcd members, losing 2 simultaneously means the remaining member cannot commit any writes — the cluster becomes read-only at best, and typically the API server itself starts refusing most requests since it can’t persist changes:

kubectl get pods

Error from server: etcdserver: request timed out

Recovering from a full quorum loss generally requires either restoring from an etcd snapshot backup onto fresh members, or — if at least one member is intact and its data directory is salvageable — using etcdctl‘s disaster-recovery procedure to force a new cluster from the surviving member’s data. Neither path is fast, which is precisely why running an odd number of members (tolerating the loss of a minority) is the actual defense, not a recovery procedure to lean on after the fact.

Certificate Rotation in an HA Cluster

kubeadm-managed certificates expire (typically after one year by default), and in an HA setup, rotation needs to happen consistently across every control-plane node rather than just the one an administrator happens to be logged into:

kubeadm certs check-expiration

kubeadm certs renew all
sudo systemctl restart kubelet

This needs to be repeated on each control-plane node individually — a step that’s easy to do once and then forget to repeat on the other two nodes, silently leaving the cluster in a state where only one control-plane node has current certificates.

Managed Kubernetes: HA Without the Manual Setup

Everything described above is what’s happening under the hood, but it’s worth being direct about the practical alternative: EKS, GKE, and AKS all provide a managed control plane where the cloud provider handles etcd replication, API server scaling, and control-plane certificate rotation automatically, with HA effectively built in rather than something the platform team assembles by hand:

eksctl create cluster --name production --nodes 3 --region us-east-1
gcloud container clusters create production --num-nodes 3 --region us-central1
az aks create --resource-group myRG --name production --node-count 3 --zones 1 2 3

For the overwhelming majority of teams, this is the more sensible default — the manual kubeadm-based HA setup described in this article is genuinely valuable to understand (both for on-premises deployments where no managed offering exists, and for understanding what’s actually happening behind a managed control plane), but it’s rarely the right choice to build and operate by hand when a managed equivalent is available and meets the organization’s compliance requirements.

When Self-Managed HA Is Still the Right Choice

There are legitimate reasons to run the manual setup described here rather than a managed offering: air-gapped or on-premises environments with no cloud provider available at all, strict data-sovereignty requirements that preclude any managed control plane, or specialized hardware requirements (GPU clusters with custom scheduling needs, for instance) that managed offerings don’t yet support well. In those cases, everything covered above — odd-numbered etcd quorum, load-balanced API servers, tested failure drills, disciplined certificate rotation, and automated snapshot backups — becomes the team’s own direct responsibility rather than something abstracted away by a cloud provider’s SLA.

Common Mistakes

Summary

A highly available Kubernetes cluster removes single points of failure from the control plane itself: multiple etcd members for quorum-based consensus, multiple API server instances behind a load balancer, and leader-elected controllers that fail over automatically. Getting this right at cluster-build time is far cheaper than retrofitting it after a 3 a.m. outage caused by one control-plane node going down.

References

Exit mobile version