The first time I lost data on a stateful workload — a Postgres instance running on a PersistentVolume with no snapshot strategy behind it — I learned the hard way that “the cloud provider handles backups” is not actually a plan. Kubernetes has had native volume snapshot support for a while now, and once I actually understood how the snapshot API objects fit together, backing up stateful workloads stopped being scary. This article covers everything from the underlying architecture to production-ready CronJob-driven snapshot automation.
Why Volume Snapshots Matter
Persistent Volumes (PVs) in Kubernetes back long-lived data — databases, message queues, file storage. Unlike stateless pods that can be rescheduled freely, losing a PV means losing data. Volume snapshots give you point-in-time, storage-level backups that you can restore from or clone into new volumes, without needing application-level backup tooling for every single workload.
The Kubernetes Volume Snapshot Architecture
Volume snapshotting is implemented through the Container Storage Interface (CSI), not through core Kubernetes APIs directly. There are three core objects:
- VolumeSnapshotClass — cluster-scoped, defines which CSI driver handles snapshots and the deletion policy, analogous to StorageClass for PVs.
- VolumeSnapshot — namespaced, the user-facing request for a snapshot of a specific PVC.
- VolumeSnapshotContent — cluster-scoped, the actual bound snapshot resource, analogous to a PV — usually created automatically when you create a VolumeSnapshot dynamically.
Internally, the external-snapshotter sidecar (running alongside the CSI driver) watches VolumeSnapshot objects, calls the CSI driver’s CreateSnapshot gRPC method, and the storage backend (EBS, GCE PD, Ceph RBD, Portworx, etc.) performs the actual block-level or filesystem-level snapshot.
VolumeSnapshot (user request)
│
▼
external-snapshotter controller
│
▼
CSI Driver (CreateSnapshot RPC)
│
▼
VolumeSnapshotContent (bound result)
│
▼
Storage backend snapshot (EBS snapshot, PD snapshot, etc.)
Prerequisites
Before you can create snapshots, your cluster needs:
- A CSI driver that supports the snapshot feature (most major ones do:
ebs-csi-driver,pd-csi-driver,azuredisk-csi-driver,ceph-csi). - The CSI Snapshotter CRDs and controller installed.
Install the snapshot CRDs and controller if they aren’t already present:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-7.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-7.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-7.0/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-7.0/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-7.0/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml
Verify:
kubectl get pods -n kube-system | grep snapshot-controller
kubectl get crd | grep snapshot.storage.k8s.io
Creating a VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: csi-snapclass
driver: ebs.csi.aws.com # swap for your CSI driver name
deletionPolicy: Delete
deletionPolicy works the same way as StorageClass’s reclaim policy — Delete removes the underlying storage snapshot when the VolumeSnapshotContent is deleted; Retain keeps it.
kubectl apply -f snapclass.yaml
kubectl get volumesnapshotclass
Taking a Manual Snapshot
Assume I have an existing PVC backing a database:
kubectl get pvc -n production
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
postgres-pvc Bound pvc-abc123 20Gi RWO gp3
Now create a VolumeSnapshot referencing that PVC:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-snapshot-2026-08-02
namespace: production
spec:
volumeSnapshotClassName: csi-snapclass
source:
persistentVolumeClaimName: postgres-pvc
kubectl apply -f postgres-snapshot.yaml
kubectl get volumesnapshot -n production
NAME READYTOUSE SOURCEPVC SNAPSHOTCLASS RESTORESIZE AGE
postgres-snapshot-2026-08-02 true postgres-pvc csi-snapclass 20Gi 45s
READYTOUSE: true means the storage backend has confirmed the snapshot completed and it’s restorable.
Restoring a Volume from a Snapshot
Restoring creates a new PVC whose dataSource points at the VolumeSnapshot:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-restored-pvc
namespace: production
spec:
storageClassName: gp3
dataSource:
name: postgres-snapshot-2026-08-02
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
kubectl apply -f postgres-restored-pvc.yaml
kubectl get pvc postgres-restored-pvc -n production
Mount this new PVC into a fresh pod (or point your StatefulSet at it) and you have a working restore — critically, without downtime on the original volume, since the snapshot operation doesn’t require unmounting the source.
Automating Snapshots with CronJobs
Manual snapshots don’t scale. I automate this with a CronJob that uses a service account with permission to create VolumeSnapshot objects on a schedule:
apiVersion: v1
kind: ServiceAccount
metadata:
name: snapshot-automation
namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: snapshot-creator
namespace: production
rules:
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["create", "get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: snapshot-creator-binding
namespace: production
subjects:
- kind: ServiceAccount
name: snapshot-automation
namespace: production
roleRef:
kind: Role
name: snapshot-creator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgres-snapshot-cron
namespace: production
spec:
schedule: "0 2 * * *" # daily at 2 AM
jobTemplate:
spec:
template:
spec:
serviceAccountName: snapshot-automation
containers:
- name: snapshotter
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
DATE=$(date +%Y%m%d-%H%M%S)
cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: postgres-snapshot-$DATE
namespace: production
spec:
volumeSnapshotClassName: csi-snapclass
source:
persistentVolumeClaimName: postgres-pvc
EOF
restartPolicy: OnFailure
Pair this with a retention job that deletes snapshots older than N days to avoid unbounded storage cost growth.
Monitoring Snapshots
I always add a check that alerts if a scheduled snapshot’s readyToUse field stays false for too long — usually indicates the CSI driver or storage backend is throttled or misconfigured. This can be scraped via a small script exporting a custom metric, or checked through kubectl get volumesnapshot -o json in a health-check Job.
Cloning Volumes Directly (Without a Snapshot)
Related to snapshots but worth distinguishing: Kubernetes also supports PVC cloning, which creates a new volume as a direct copy of an existing PVC without going through the VolumeSnapshot object at all. This is useful for quickly spinning up a dev/test copy of production data:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-dev-clone
namespace: staging
spec:
storageClassName: gp3
dataSource:
name: postgres-pvc
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
kubectl apply -f postgres-dev-clone.yaml
kubectl get pvc postgres-dev-clone -n staging
Note that cloning requires the source and destination PVC to be in the same namespace and typically the same StorageClass — this is a meaningful limitation compared to snapshot-based restores, which can cross namespaces more flexibly since the VolumeSnapshot itself can be referenced from a different context.
Troubleshooting Failed Snapshots
VolumeSnapshot stuck with readyToUse: false. Check the VolumeSnapshotContent object it’s bound to for the actual error:
kubectl get volumesnapshot postgres-snapshot-2026-08-02 -n production -o yaml
kubectl get volumesnapshotcontent -o yaml | grep -A 5 "error"
Common causes are IAM/permission issues on the CSI driver’s service account (the controller can’t call CreateSnapshot against the cloud API), or a mismatched VolumeSnapshotClass driver name that doesn’t correspond to any registered CSI driver:
kubectl get csidriver
kubectl get volumesnapshotclass -o yaml
Restore PVC stuck in Pending. This is very often a size mismatch — many CSI drivers require the new PVC’s storage request to be greater than or equal to the snapshot’s restoreSize:
kubectl get volumesnapshot postgres-snapshot-2026-08-02 -n production -o jsonpath='{.status.restoreSize}'
Set the new PVC’s request to at least this value.
Snapshot succeeds but restored data looks inconsistent. This is almost always a crash-consistency vs application-consistency issue — the snapshot was taken mid-write without a quiesce step. For databases, always pause writes or use the database’s native backup-mode hooks before triggering the snapshot.
Security Considerations
Snapshot data is just as sensitive as the source volume — a leaked snapshot is a leaked database. A few things I enforce:
- Restrict
volumesnapshotsandvolumesnapshotcontentsRBAC verbs the same way you’d restrict access to the PVC itself. - For cloud-backed snapshots, ensure the underlying storage snapshots (EBS snapshots, GCE PD snapshots) inherit encryption settings from the source volume — don’t assume this happens automatically for every CSI driver.
- Audit snapshot deletion carefully; a
deletionPolicy: DeleteVolumeSnapshotClass means deleting the Kubernetes object also deletes your actual backup, which is easy to do by accident during a namespace cleanup.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: snapshot-viewer
namespace: production
rules:
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch"]
Production Best Practices
- Application-consistent snapshots: For databases, take snapshots after a quiesce/flush step (e.g.,
pg_start_backup()for Postgres) rather than relying purely on crash-consistent storage snapshots, unless your storage backend guarantees write-order consistency. - Cross-region copies: Native VolumeSnapshots are typically bound to the same region/zone as the source volume. For disaster recovery, replicate the underlying cloud snapshot (e.g., EBS snapshot copy) to another region using your cloud provider’s native tooling alongside Kubernetes snapshots.
- RBAC scoping: Restrict
volumesnapshotscreate/delete permissions tightly — a compromised automation account could delete your only backups. - Test restores regularly: A snapshot you’ve never restored from is a hypothesis, not a backup.
Common Mistakes
- Assuming VolumeSnapshot works without the CSI driver explicitly supporting the snapshot feature — check
kubectl get csidriverand driver docs first. - Forgetting
deletionPolicy: Retainfor critical snapshot classes, leading to accidental data loss when a namespace is deleted. - Not setting resource requests to match
restoreSizeexactly when the storage backend requires exact size matches.
Summary
Volume snapshots turn Kubernetes-native storage into something you can actually trust for stateful workloads. The CSI-based VolumeSnapshot/VolumeSnapshotContent/VolumeSnapshotClass trio maps closely to the familiar PVC/PV/StorageClass pattern, which makes it approachable once you’ve internalized the architecture. Automate the scheduling, monitor readyToUse status, and — above everything else — actually test your restores.
