For a long time, “Kubernetes is for stateless apps” was conventional wisdom, and running a database on it felt like asking for trouble. That’s changed a lot — StatefulSet, mature storage drivers, and battle-tested operators have made running stateful workloads like databases and message queues on Kubernetes a normal, supportable thing to do. In this guide I’ll walk through deploying a real stateful application (a PostgreSQL-style database) using StatefulSet, persistent storage, and a headless Service, and cover the operational realities that make stateful workloads different from stateless ones.
What Makes Stateful Different
Stateless apps (a typical web API, for example) are interchangeable — any replica can serve any request, and losing one is a non-event. Stateful apps break that assumption:
- Each replica often needs its own persistent storage, not shared or ephemeral.
- Replicas frequently need stable, predictable network identities (a database primary needs to be reliably addressable, not randomly assigned a new DNS name every restart).
- Startup and shutdown order often matters (a replica joining a cluster needs the primary to already be up).
- Scaling isn’t just “add more identical copies” — it often means reconfiguring cluster membership.
StatefulSet addresses the first two directly; the rest usually needs application-level logic or an operator.
Step 1: Understanding StatefulSet Basics
Unlike a Deployment, a StatefulSet:
- Assigns each Pod a stable, ordinal name (
myapp-0,myapp-1,myapp-2), not a random suffix. - Creates Pods in order (0, then 1, then 2) and terminates them in reverse order by default.
- Pairs with a headless Service to give each Pod a stable DNS name (
myapp-0.myapp.default.svc.cluster.local). - Uses
volumeClaimTemplatesto give each Pod its own dedicatedPersistentVolumeClaim, which survives Pod rescheduling.
Step 2: Create a Headless Service
# postgres-headless-service.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: postgres
spec:
clusterIP: None
selector:
app: postgres
ports:
- port: 5432
name: postgres
clusterIP: None is what makes this “headless” — instead of load-balancing, DNS returns the individual Pod IPs directly, which is exactly what you want for addressing a specific replica.
Step 3: Define the StatefulSet
# postgres-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 3
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16
ports:
- containerPort: 5432
name: postgres
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: "1"
memory: 2Gi
readinessProbe:
exec:
command: ["pg_isready", "-U", "postgres"]
initialDelaySeconds: 10
periodSeconds: 10
volumeClaimTemplates:
- metadata:
name: postgres-storage
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 20Gi
Create the required Secret first:
kubectl create secret generic postgres-secret --from-literal=password='ChangeMeInProduction!'
kubectl apply -f postgres-headless-service.yaml
kubectl apply -f postgres-statefulset.yaml
Step 4: Watch the Ordered Rollout
kubectl get pods -l app=postgres --watch
NAME READY STATUS RESTARTS AGE
postgres-0 0/1 ContainerCreating 0 5s
postgres-0 1/1 Running 0 20s
postgres-1 0/1 ContainerCreating 0 22s
postgres-1 1/1 Running 0 40s
postgres-2 0/1 ContainerCreating 0 42s
postgres-2 1/1 Running 0 60s
Notice postgres-1 doesn’t even start creating until postgres-0 is Running and Ready — that ordering guarantee is central to how StatefulSet behaves by default (podManagementPolicy: OrderedReady).
Check the PersistentVolumeClaims created automatically, one per Pod:
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
postgres-storage-postgres-0 Bound pvc-a1b2c3 20Gi RWO fast-ssd
postgres-storage-postgres-1 Bound pvc-d4e5f6 20Gi RWO fast-ssd
postgres-storage-postgres-2 Bound pvc-g7h8i9 20Gi RWO fast-ssd
Step 5: Verify Stable Network Identity
kubectl run debug --rm -it --image=busybox -- nslookup postgres-0.postgres.default.svc.cluster.local
Name: postgres-0.postgres.default.svc.cluster.local
Address 1: 10.244.1.15
Even if postgres-0 is rescheduled to a different node, it comes back with the same name and reattaches to the same PVC — the identity and the data both persist.
Step 6: Scaling a StatefulSet
kubectl scale statefulset postgres --replicas=5
New Pods (postgres-3, postgres-4) are created in order, each getting a fresh PVC. Note that scaling down does not delete the PVCs by default — Kubernetes leaves them in place intentionally, so scaling back up later reattaches the same data rather than starting fresh. Clean these up manually if you truly want to discard them:
kubectl delete pvc postgres-storage-postgres-4
Step 7: Handling Updates Carefully
StatefulSet supports RollingUpdate by default, updating Pods in reverse ordinal order (highest number first), which for a typical primary-replica database setup means replicas get updated before the primary:
spec:
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 0
The partition field is worth knowing — setting it to a number means only Pods with an ordinal greater than or equal to that number get updated, letting you stage a canary-style update of just the highest-numbered replica before rolling out to the rest.
Step 8: Consider an Operator for Real Production Databases
Hand-rolling a StatefulSet for PostgreSQL, as above, works for learning and simple cases, but for production I’d strongly lean toward a purpose-built operator — something like CloudNativePG, Zalando’s Postgres Operator, or the Bitnami PostgreSQL HA chart — because they handle replication setup, automated failover, backups, and point-in-time recovery, none of which a bare StatefulSet gives you on its own:
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm install cnpg-operator cnpg/cloudnative-pg -n cnpg-system --create-namespace
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: myapp-db
spec:
instances: 3
storage:
size: 20Gi
storageClass: fast-ssd
This single CR gives you automated primary election, streaming replication, and backup scheduling — things that would take substantial custom logic to build on a raw StatefulSet.
Storage and Disaster Recovery Considerations
- Use a
StorageClassbacked by real replicated, durable storage (cloud provider block storage, Ceph, Longhorn) — not localhostPathvolumes, which don’t survive node loss. - Set
reclaimPolicy: Retainon yourStorageClassfor stateful workloads so accidental PVC deletion doesn’t immediately destroy the underlying volume. - Schedule regular backups external to the cluster (e.g., to object storage) — PVC snapshots alone aren’t a full disaster recovery strategy if your entire cluster or cloud region goes down.
- Test restore procedures regularly; an untested backup is a hypothesis, not a plan.
Common Mistakes
- Using a
Deploymentinstead of aStatefulSetfor anything with per-replica state, then being surprised when a rescheduled Pod loses its data or identity. - Forgetting the headless Service, which breaks the stable DNS naming
StatefulSetdepends on. - Assuming scaling down deletes data — it doesn’t, which can be a surprise cost (unused PVCs sitting around) or a lifesaver depending on your expectations.
- Running a single-replica “StatefulSet” as if it provides high availability — it doesn’t; replication and failover logic still has to come from the app or an operator.
Anti-Affinity for Real Fault Tolerance
Running three replicas of a database means little for availability if all three land on the same physical node, which then has a single point of failure regardless of what the StatefulSet spec claims about replica count. Pod anti-affinity spreads replicas across nodes (and, ideally, availability zones):
spec:
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values: ["postgres"]
topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: postgres
Using requiredDuringSchedulingIgnoredDuringExecution here (rather than preferred) means the scheduler will refuse to place two replicas on the same node at all, rather than merely preferring not to — worth the tradeoff of slightly stricter scheduling requirements for genuinely critical stateful workloads.
PodDisruptionBudget for Stateful Workloads
Voluntary disruptions — node drains for maintenance, cluster upgrades, Cluster Autoscaler consolidating nodes — can just as easily take down database replicas as any other workload. A PodDisruptionBudget prevents Kubernetes from voluntarily evicting enough replicas at once to break quorum:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: postgres-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: postgres
With minAvailable: 2 on a 3-replica set, Kubernetes will block a node drain (or any other voluntary disruption) that would drop availability below 2 replicas, forcing whoever’s performing maintenance to wait or address the situation rather than silently taking your database below quorum.
Backup and Restore Workflow in Practice
A StatefulSet (or an operator built on top of one) gives you data durability against Pod rescheduling, but it doesn’t protect against application-level mistakes — an accidental DROP TABLE, a bad migration, ransomware, or a full cluster loss. A practical backup workflow I’ve used for PostgreSQL running via CloudNativePG:
apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
name: myapp-db-backup
spec:
schedule: "0 2 * * *"
backupOwnerReference: self
cluster:
name: myapp-db
This schedules a nightly backup to object storage (configured separately on the Cluster resource), independent of the PVC snapshots your cloud provider might also offer. I treat these as complementary, not redundant: PVC snapshots are fast to restore from for routine node-level failures, while object-storage backups (ideally in a different region or account) are what actually protects you against a full cluster or region loss.
Testing the restore path matters as much as the backup itself:
kubectl apply -f restore-cluster.yaml # points at the backup, spins up a fresh Cluster
kubectl get pods -l cnpg.io/cluster=myapp-db-restored
I schedule a quarterly “restore drill” on any team I work with — actually restoring a backup into a scratch namespace and verifying the data is intact and queryable — because an untested backup strategy has a way of failing exactly when you need it most.
Monitoring Stateful Workloads
Beyond generic Pod health, stateful workloads need domain-specific monitoring — replication lag, connection pool saturation, disk usage trending toward the PVC limit:
pg_replication_lag_seconds > 10
kubelet_volume_stats_used_bytes{persistentvolumeclaim=~"postgres-storage.*"}
/ kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~"postgres-storage.*"} > 0.85
The second query — PVC usage climbing above 85% — deserves its own dedicated alert distinct from general resource alerts, since a database that runs out of disk space fails in ways that are considerably more disruptive to recover from than a stateless Pod simply restarting.
Summary
StatefulSet gives Kubernetes the primitives stateful applications need: stable identities, ordered deployment and scaling, and per-replica persistent storage via volumeClaimTemplates. For genuinely production-grade databases, though, pairing StatefulSet concepts with a dedicated operator will save you from reimplementing replication, failover, and backup logic yourself. Either way, understanding how StatefulSet differs from Deployment is the foundation for running any stateful workload reliably on Kubernetes.
