A single control-plane node is fine for a home lab. It is not fine for anything that matters, because the moment that one node goes down, you lose the ability to schedule, scale, or even query the state of your cluster — even if every application Pod keeps running untouched on the data plane. Building a highly available cluster means removing every single point of failure from the control plane, not just the workloads running on top of it.
What “Highly Available” Actually Means Here
There are two separate failure domains to address:
- Control plane HA — multiple API servers, multiple etcd members, no single node whose failure breaks cluster management.
- Workload HA — multiple replicas of applications, spread across failure domains, surviving node loss (covered by PodAntiAffinity, topology spread, and PodDisruptionBudgets elsewhere).
This article focuses primarily on the first, since it’s the part people most often get wrong or skip entirely.
Control Plane Architecture
A production HA control plane needs, at minimum:
- 3 (or 5) etcd members — an odd number, since etcd uses Raft consensus and needs a majority quorum to stay writable. 3 members tolerate 1 failure; 5 tolerate 2.
- 3 API server instances, typically co-located with etcd on the same nodes (stacked topology) or on separate nodes (external etcd topology).
- A load balancer in front of the API servers, since
kubectland kubelets need one stable endpoint regardless of which API server instance is actually alive.
┌────────────────────┐
│ Load Balancer │
│ (kube-apiserver) │
└─────────┬──────────┘
┌─────────────┼─────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ CP Node1│ │ CP Node2│ │ CP Node3│
│apiserver│ │apiserver│ │apiserver│
│ etcd │ │ etcd │ │ etcd │
│scheduler│ │scheduler│ │scheduler│
│ ctrl-mgr│ │ ctrl-mgr│ │ ctrl-mgr│
└─────────┘ └─────────┘ └─────────┘
Only one kube-scheduler and one kube-controller-manager are ever active at a time — they use leader election, so running 3 replicas of each is about failover speed, not concurrent operation.
Bootstrapping with kubeadm
Set up a load balancer first (HAProxy example):
# /etc/haproxy/haproxy.cfg
frontend kubernetes-frontend
bind *:6443
mode tcp
default_backend kubernetes-backend
backend kubernetes-backend
mode tcp
balance roundrobin
server cp1 10.0.1.11:6443 check
server cp2 10.0.1.12:6443 check
server cp3 10.0.1.13:6443 check
Initialize the first control-plane node, pointing at the load balancer’s address:
kubeadm init \
--control-plane-endpoint "lb.example.com:6443" \
--upload-certs \
--pod-network-cidr=192.168.0.0/16
Output includes two join commands — save both:
You can now join any number of control-plane nodes by running:
kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1234... \
--control-plane --certificate-key 5678...
You can now join any number of worker nodes by running:
kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1234...
Join the second and third control-plane nodes:
kubeadm join lb.example.com:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:1234... \
--control-plane --certificate-key 5678...
Join worker nodes with the second command (no --control-plane flag).
Verifying the Control Plane
kubectl get nodes
Output:
NAME STATUS ROLES AGE VERSION
cp1 Ready control-plane 10m v1.30.2
cp2 Ready control-plane 8m v1.30.2
cp3 Ready control-plane 7m v1.30.2
node1 Ready <none> 5m v1.30.2
node2 Ready <none> 5m v1.30.2
Check etcd cluster health directly:
kubectl -n kube-system exec etcd-cp1 -- etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health --cluster
Output:
https://10.0.1.11:2379 is healthy: successfully committed proposal
https://10.0.1.12:2379 is healthy: successfully committed proposal
https://10.0.1.13:2379 is healthy: successfully committed proposal
Networking (CNI)
A CNI plugin has to be installed before nodes report Ready. Calico is a common production choice for its NetworkPolicy support:
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml
RBAC for Cluster Administration
Once the cluster is up, scope access properly rather than distributing the admin kubeconfig broadly:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: platform-team-admin
subjects:
- kind: Group
name: platform-team
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
Worker Node HA Considerations
Beyond the control plane, worker-side HA involves spreading nodes across availability zones and using cluster autoscaling to replace lost capacity automatically:
apiVersion: v1
kind: Node
metadata:
labels:
topology.kubernetes.io/zone: us-east-1a
Application-level HA (replica count, PodAntiAffinity, PodDisruptionBudgets) then layers on top of this zone-aware node topology.
Monitoring the Control Plane
kubectl get componentstatuses
kubectl -n kube-system get pods -o wide
For ongoing visibility, scrape control-plane metrics into the Grafana/Prometheus stack:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: kube-apiserver
namespace: monitoring
spec:
selector:
matchLabels:
component: apiserver
endpoints:
- port: https
scheme: https
tlsConfig:
insecureSkipVerify: true
Backup and Disaster Recovery
etcd is the single source of truth for cluster state — losing it without a backup means losing the cluster’s entire configuration, not just workloads:
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%F).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
Automate this with a CronJob or an external backup tool, and periodically test restoring it — an untested backup is not a real backup.
Testing HA Before You Need It
An HA control plane that has never actually been tested against a real node failure is a hypothesis, not a guarantee. Deliberately testing failure scenarios in a staging environment (never production, for obvious reasons) builds real confidence:
# Simulate losing one control-plane node
kubectl drain cp2 --ignore-daemonsets --delete-emptydir-data
sudo systemctl stop kubelet kube-apiserver etcd
# Confirm the cluster is still fully functional
kubectl get nodes
kubectl create deployment test --image=nginx
kubectl scale deployment test --replicas=5
If everything above still works with cp2 down, the load balancer and etcd quorum are doing their job correctly. Bring the node back and confirm it rejoins cleanly:
sudo systemctl start etcd kube-apiserver kubelet
kubectl get nodes
kubectl -n kube-system exec etcd-cp1 -- etcdctl endpoint health --cluster
Running this drill periodically — not just once at initial setup — catches configuration drift before an actual outage does. A control plane that was HA six months ago after a manual node replacement or a certificate rotation isn’t guaranteed to still be HA today without a periodic check.
What Happens When etcd Loses Quorum
Understanding the actual failure mode matters as much as preventing it. With 3 etcd members, losing 2 simultaneously means the remaining member cannot commit any writes — the cluster becomes read-only at best, and typically the API server itself starts refusing most requests since it can’t persist changes:
kubectl get pods
Error from server: etcdserver: request timed out
Recovering from a full quorum loss generally requires either restoring from an etcd snapshot backup onto fresh members, or — if at least one member is intact and its data directory is salvageable — using etcdctl‘s disaster-recovery procedure to force a new cluster from the surviving member’s data. Neither path is fast, which is precisely why running an odd number of members (tolerating the loss of a minority) is the actual defense, not a recovery procedure to lean on after the fact.
Certificate Rotation in an HA Cluster
kubeadm-managed certificates expire (typically after one year by default), and in an HA setup, rotation needs to happen consistently across every control-plane node rather than just the one an administrator happens to be logged into:
kubeadm certs check-expiration
kubeadm certs renew all
sudo systemctl restart kubelet
This needs to be repeated on each control-plane node individually — a step that’s easy to do once and then forget to repeat on the other two nodes, silently leaving the cluster in a state where only one control-plane node has current certificates.
Managed Kubernetes: HA Without the Manual Setup
Everything described above is what’s happening under the hood, but it’s worth being direct about the practical alternative: EKS, GKE, and AKS all provide a managed control plane where the cloud provider handles etcd replication, API server scaling, and control-plane certificate rotation automatically, with HA effectively built in rather than something the platform team assembles by hand:
eksctl create cluster --name production --nodes 3 --region us-east-1
gcloud container clusters create production --num-nodes 3 --region us-central1
az aks create --resource-group myRG --name production --node-count 3 --zones 1 2 3
For the overwhelming majority of teams, this is the more sensible default — the manual kubeadm-based HA setup described in this article is genuinely valuable to understand (both for on-premises deployments where no managed offering exists, and for understanding what’s actually happening behind a managed control plane), but it’s rarely the right choice to build and operate by hand when a managed equivalent is available and meets the organization’s compliance requirements.
When Self-Managed HA Is Still the Right Choice
There are legitimate reasons to run the manual setup described here rather than a managed offering: air-gapped or on-premises environments with no cloud provider available at all, strict data-sovereignty requirements that preclude any managed control plane, or specialized hardware requirements (GPU clusters with custom scheduling needs, for instance) that managed offerings don’t yet support well. In those cases, everything covered above — odd-numbered etcd quorum, load-balanced API servers, tested failure drills, disciplined certificate rotation, and automated snapshot backups — becomes the team’s own direct responsibility rather than something abstracted away by a cloud provider’s SLA.
Common Mistakes
- Running a single control-plane node “temporarily” and never getting around to converting it to HA before it matters.
- Using an even number of etcd members (e.g. 2 or 4), which doesn’t actually improve fault tolerance over an odd number and can create split-brain-adjacent quorum issues.
- Pointing
kubectl/kubelets directly at one control-plane node’s IP instead of the load balancer, silently reintroducing a single point of failure. - Never testing an etcd restore, discovering during a real disaster that the backup process was broken all along.
Summary
A highly available Kubernetes cluster removes single points of failure from the control plane itself: multiple etcd members for quorum-based consensus, multiple API server instances behind a load balancer, and leader-elected controllers that fail over automatically. Getting this right at cluster-build time is far cheaper than retrofitting it after a 3 a.m. outage caused by one control-plane node going down.
References
- Kubernetes HA topology options: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/
- kubeadm HA cluster guide: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
- etcd documentation: https://etcd.io/docs/
- Calico documentation: https://docs.tigera.io/calico/latest/about/
