Deployments are great until you need something Kubernetes’ default “cattle, not pets” philosophy actively fights against: stable network identities, ordered startup, and storage that follows a specific pod instance across restarts. That’s the exact gap StatefulSets fill, and if you’ve ever tried running a database or a Kafka cluster on a plain Deployment, you already know why they exist. Let’s go through StatefulSets thoroughly, from the mechanics to a full production example on AWS EKS.
Why Deployments Aren’t Enough for Stateful Workloads
A Deployment’s pods are interchangeable — they get random name suffixes, no guaranteed identity, and if backed by a shared PVC, multiple replicas would fight over the same volume (which is disallowed for RWO storage anyway). For applications like PostgreSQL replicas, Kafka brokers, Elasticsearch nodes, or ZooKeeper ensembles, each instance needs:
- A stable, predictable network identity (e.g.,
kafka-0,kafka-1,kafka-2— not random hashes) - Its own dedicated, persistent storage that follows it across rescheduling
- Ordered, sequential startup and shutdown (so a database primary starts before replicas try to connect to it)
StatefulSet provides exactly these three guarantees.
How StatefulSets Work Internally
A StatefulSet requires a headless Service (clusterIP: None) to provide network identity. Instead of load-balancing across pods like a normal Service, a headless Service creates DNS records for each individual pod: <pod-name>.<service-name>.<namespace>.svc.cluster.local.
Pods are named deterministically: <statefulset-name>-0, <statefulset-name>-1, <statefulset-name>-2, and so on. By default, pods are created and terminated in strict ordinal order — pod 1 won’t be created until pod 0 is Running and Ready, and on scale-down, the highest ordinal is removed first.
Each pod gets its own PVC generated from a volumeClaimTemplate, named <volume-name>-<pod-name>. Critically, if a pod is deleted and recreated (e.g., after a node failure), it reattaches to the same PVC — the data isn’t lost or reshuffled between replicas.
A Complete PostgreSQL StatefulSet Example
Let’s build a realistic 3-node example — for illustration I’ll model it loosely on a primary/replica setup, though in practice you’d typically use a purpose-built operator like Zalando’s postgres-operator or CloudNativePG for real Postgres HA. Understanding the raw StatefulSet mechanics first makes those operators much less mysterious.
Headless Service for stable network identity:
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: database
labels:
app: postgres
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
name: postgres
The StatefulSet itself:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
namespace: database
spec:
serviceName: postgres
replicas: 3
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
terminationGracePeriodSeconds: 30
containers:
- name: postgres
image: postgres:16
ports:
- containerPort: 5432
name: postgres
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
readinessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
initialDelaySeconds: 10
periodSeconds: 10
livenessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
initialDelaySeconds: 30
periodSeconds: 20
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3
resources:
requests:
storage: 50Gi
kubectl apply -f postgres-headless-service.yaml -f postgres-statefulset.yaml
kubectl get statefulset postgres -n database
NAME READY AGE
postgres 3/3 4m
kubectl get pods -n database -l app=postgres -o wide
NAME READY STATUS RESTARTS AGE NODE
postgres-0 1/1 Running 0 4m ip-10-0-1-15
postgres-1 1/1 Running 0 3m ip-10-0-2-22
postgres-2 1/1 Running 0 2m ip-10-0-3-31
Notice they came up sequentially — postgres-1 didn’t start until postgres-0 was ready, exactly the ordering guarantee we need if postgres-0 acts as primary and the others need it available first.
Check the PVCs — one per pod, following the naming pattern:
kubectl get pvc -n database
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
data-postgres-0 Bound pvc-a1b2... 50Gi RWO gp3
data-postgres-1 Bound pvc-c3d4... 50Gi RWO gp3
data-postgres-2 Bound pvc-e5f6... 50Gi RWO gp3
DNS and Stable Network Identity
Each pod is individually addressable:
kubectl run -it --rm debug --image=busybox --restart=Never -n database -- \
nslookup postgres-0.postgres.database.svc.cluster.local
Name: postgres-0.postgres.database.svc.cluster.local
Address 1: 10.0.1.45 postgres-0.postgres.database.svc.cluster.local
This is what makes StatefulSets viable for distributed systems that need to reference specific peers by name — Kafka’s advertised.listeners, Elasticsearch’s discovery.seed_hosts, or a Postgres replica’s primary_conninfo can all point at a stable, predictable hostname rather than a rotating IP.
Pod Management Policies
- OrderedReady (default) — pods are created/deleted one at a time, in order, waiting for
Readybetween each. Safer, slower. - Parallel — all pods are created/deleted simultaneously. Faster, but only appropriate when your application doesn’t depend on startup ordering (e.g., a Cassandra ring, which handles concurrent node joins natively).
spec:
podManagementPolicy: Parallel
Rolling Updates with Partitions
StatefulSet rolling updates go in reverse ordinal order (highest number first) by default — the opposite of pod creation order. This matters for databases: you typically want to update replicas before the primary.
The partition field lets you stage canary-style updates across a StatefulSet:
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 2
With partition: 2 on a 3-replica StatefulSet, only pods with ordinal >= 2 (i.e., just postgres-2) get updated when you change the pod template. Lower-ordinal pods are left untouched. This lets you validate an update on one replica before lowering the partition to roll it out further:
kubectl patch statefulset postgres -n database --type='json' \
-p='[{"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value": 0}]'
Scaling
kubectl scale statefulset postgres -n database --replicas=5
New pods (postgres-3, postgres-4) get new PVCs automatically provisioned. Scaling down removes the highest-ordinal pods first, but PVCs are not deleted automatically — this is intentional, protecting you from accidental data loss, but it means you need to clean up orphaned PVCs manually if you’re sure you no longer need that data:
kubectl get pvc -n database
kubectl delete pvc data-postgres-4 -n database
Production Pattern: Use an Operator for Real Databases
While hand-rolling a StatefulSet teaches you the mechanics, running real production databases on Kubernetes almost always benefits from a purpose-built Operator that understands the application’s specific failover, backup, and replication logic — a raw StatefulSet has no idea what “promote a replica to primary” means for Postgres.
# Example: CloudNativePG operator
kubectl apply --server-side -f \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.23.1.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: postgres-cluster
namespace: database
spec:
instances: 3
storage:
size: 50Gi
storageClass: gp3
bootstrap:
initdb:
database: appdb
owner: appuser
Under the hood, CloudNativePG still uses a StatefulSet-like pattern (technically it manages individual Pods with PVCs directly for finer control), but exposes proper failover automation, backup scheduling to S3, and connection pooling on top — all things you’d otherwise have to build yourself.
Monitoring StatefulSets
kubectl get statefulset postgres -n database -o wide
kubectl describe statefulset postgres -n database
kubectl rollout status statefulset/postgres -n database
For Prometheus-based monitoring (see the companion Prometheus article), useful metrics from kube-state-metrics include:
kube_statefulset_status_replicas_ready{statefulset="postgres"}
kube_statefulset_status_current_revision != kube_statefulset_status_update_revision
That second query flags StatefulSets mid-rollout or stuck on a partial update.
Troubleshooting
Pod stuck waiting on a prior ordinal:
kubectl describe pod postgres-1 -n database
If postgres-0 never became Ready, postgres-1 will never be created under OrderedReady — check postgres-0‘s readiness probe and logs first.
PVC stuck Pending after scale-up:
Check the same AZ-mismatch and StorageClass issues covered in the Persistent Volume Claims article — WaitForFirstConsumer binding mode applies here too.
Stale data after pod rescheduling to a different node:
This shouldn’t happen with EBS-backed PVCs (the volume follows the pod), but if you’re using hostPath or local storage for a StatefulSet, verify node affinity is correctly pinning pods to nodes with local data — this is a common misconfiguration when moving from cloud block storage to local NVMe setups.
Common Mistakes
- Deleting a StatefulSet’s PVCs thinking a plain
kubectl delete -fon the StatefulSet cleans them up — it doesn’t, by design, but this surprises people in both directions (unexpected retention, and unexpected assumption of retention). - Assuming
Parallelpod management is always faster and safer — for genuinely ordering-sensitive applications it can cause split-brain or bootstrap race conditions. - Forgetting the headless Service — without
clusterIP: None, you don’t get the stable per-pod DNS entries that make StatefulSets useful in the first place. - Hand-rolling primary/replica failover logic instead of using a mature operator for the specific database engine in question.
Best Practices
- Use a dedicated operator (CloudNativePG, Strimzi for Kafka, ECK for Elasticsearch) for production stateful workloads rather than raw StatefulSets whenever one exists for your technology.
- Set resource requests/limits generously and realistically for stateful workloads — OOMKills on a database are far more disruptive than on a stateless API pod.
- Use
partitionfor staged, validated rollouts on anything storing critical data. - Monitor PVC usage proactively — full disks on a stateful workload are a much worse failure mode than a stateless pod running out of memory.
Summary
StatefulSets give Kubernetes workloads the three things Deployments deliberately don’t provide: stable network identity via headless Services, ordered startup/shutdown, and per-replica persistent storage that survives rescheduling. They’re the right primitive for databases, message queues, and any distributed system where individual instance identity matters. For real production database workloads, layer a purpose-built operator on top rather than reinventing failover and backup logic yourself — but understanding the raw StatefulSet mechanics makes those operators far less of a black box.
