Containers are meant to be ephemeral — kill one, start another, nobody should notice. That’s a great property until your application actually needs to remember something between restarts: uploaded files, a database’s data directory, a cache that’s expensive to rebuild. That’s exactly what Kubernetes’ storage abstractions — PersistentVolume and PersistentVolumeClaim — are built for. In this guide I’ll break down how these objects relate to each other, walk through both static and dynamic provisioning, and cover the access modes and reclaim policies that trip people up most.
The Core Abstractions
Kubernetes deliberately separates how storage is provisioned from how an application asks for storage:
- PersistentVolume (PV) — a piece of actual storage in the cluster, provisioned either manually by an admin or dynamically by a
StorageClass. It exists independently of any particular Pod. - PersistentVolumeClaim (PVC) — a request for storage made by a user/application, specifying size and access mode. Kubernetes binds a PVC to a matching PV.
- StorageClass — describes a “class” of storage (e.g., SSD-backed, HDD-backed, a specific cloud disk type) and enables dynamic provisioning, so PVs get created on-demand rather than pre-created by hand.
Think of a StorageClass as a menu, a PVC as an order placed against that menu, and the resulting PV as the dish that gets delivered and bound exclusively to whoever ordered it.
Step 1: Check Available StorageClasses
On any managed cloud cluster, at least one default StorageClass usually already exists:
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE DEFAULT
standard (default) kubernetes.io/gce-pd Delete WaitForFirstConsumer true
fast-ssd pd.csi.storage.gke.io Retain WaitForFirstConsumer false
If none exists (common on bare-metal or local clusters), you’ll need to install a CSI driver appropriate for your storage backend, or use static provisioning instead.
Step 2: Dynamic Provisioning with a PVC (the Common Path)
Most of the time you won’t manually create a PersistentVolume at all — you just create a PersistentVolumeClaim, and the StorageClass‘s provisioner creates the matching PV automatically:
# myapp-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 10Gi
kubectl apply -f myapp-pvc.yaml
kubectl get pvc myapp-data
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
myapp-data Bound pvc-3f9a2b1c-8e4d-4a6f-9c2e-1234567890ab 10Gi RWO fast-ssd
Note the VOLUME column — that’s the automatically created PersistentVolume bound to this claim. You can inspect it:
kubectl get pv pvc-3f9a2b1c-8e4d-4a6f-9c2e-1234567890ab
Step 3: Mount the PVC in a Pod
# myapp-with-storage.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myrepo/myapp:1.0.0
volumeMounts:
- name: data
mountPath: /var/lib/myapp/data
volumes:
- name: data
persistentVolumeClaim:
claimName: myapp-data
kubectl apply -f myapp-with-storage.yaml
kubectl exec -it deploy/myapp -- df -h /var/lib/myapp/data
Write a test file, delete the Pod, and confirm data survives:
kubectl exec -it deploy/myapp -- sh -c "echo hello > /var/lib/myapp/data/test.txt"
kubectl delete pod -l app=myapp
kubectl exec -it deploy/myapp -- cat /var/lib/myapp/data/test.txt
hello
Because the new Pod (recreated by the Deployment controller) mounts the same PVC, the data persists across Pod restarts — this is the entire point.
Step 4: Understanding Access Modes
- ReadWriteOnce (RWO) — mounted read-write by a single node at a time (note: not strictly a single Pod on some CSI drivers, but functionally treat it as single-node).
- ReadOnlyMany (ROX) — mounted read-only by many nodes simultaneously.
- ReadWriteMany (RWX) — mounted read-write by many nodes simultaneously — requires a storage backend that supports it, like NFS, EFS, Azure Files, or CephFS. Most cloud block storage (EBS, GCE PD) only supports RWO.
- ReadWriteOncePod (RWOP) — a newer mode restricting mount to a single Pod, not just a single node, useful for StatefulSets needing strict exclusivity.
Picking the wrong access mode is one of the most common storage mistakes — trying to scale a Deployment with an RWO volume to multiple replicas across nodes will leave extra Pods stuck Pending because the volume can’t attach to more than one node at once.
Step 5: Static Provisioning (Manual PV Creation)
Sometimes — connecting to an existing NFS share, for example — you need to define the PersistentVolume yourself rather than relying on dynamic provisioning:
# nfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: nfs-data-pv
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 10.0.0.5
path: /exports/myapp-data
# nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-data-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
resources:
requests:
storage: 50Gi
kubectl apply -f nfs-pv.yaml
kubectl apply -f nfs-pvc.yaml
kubectl get pv nfs-data-pv
Setting storageClassName: "" on the PVC tells Kubernetes not to try dynamic provisioning and instead bind to an existing, matching PV — matching happens based on capacity, access mode, and any selector you specify.
Step 6: Reclaim Policies
This determines what happens to the underlying storage when a PVC is deleted:
- Delete (common default for dynamically provisioned volumes) — the PV and underlying storage are deleted along with the PVC. Fast, but dangerous for anything you can’t afford to lose.
- Retain — the PV survives PVC deletion, moving to a
Releasedstate, but is not automatically reusable until an admin manually cleans it up and rebinds it.
For production databases, I always override the default to Retain:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast-ssd-retain
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
parameters:
type: pd-ssd
Step 7: Resizing a PVC
Many CSI drivers support volume expansion without downtime:
kubectl patch pvc myapp-data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
kubectl get pvc myapp-data
Check that the StorageClass has allowVolumeExpansion: true — without it, this patch will be rejected.
Debugging PVC Issues
A PVC stuck in Pending:
kubectl describe pvc myapp-data
Common causes shown in the events:
Warning ProvisioningFailed 10s persistentvolume-controller
storageclass.storage.k8s.io "fast-ssd" not found
or
Warning FailedBinding 10s persistentvolume-controller
no persistent volumes available for this claim and no storage class is set
The first means a typo in storageClassName; the second means you’re relying on static provisioning but haven’t created a matching PV yet, or there’s no default StorageClass configured.
Also worth checking volumeBindingMode: WaitForFirstConsumer — with this mode (recommended for cloud environments), the PVC intentionally stays Pending until a Pod referencing it is actually scheduled, so the volume can be provisioned in the correct availability zone. Seeing “Pending” here is normal, not a bug, until a Pod shows up.
Best Practices
- Prefer dynamic provisioning with
StorageClassover manually managing PVs — it scales much better operationally. - Set
reclaimPolicy: Retainfor anything with important data;Deleteis fine for scratch or cache volumes only. - Match access modes to actual workload topology — RWO for single-writer databases, RWX only when genuinely needed (and only on backends that support it).
- Monitor PVC usage (
kube_persistentvolumeclaim_resource_requests_storage_bytesin Prometheus) so volumes don’t silently fill up. - Back up data at the application or snapshot level in addition to relying on volume durability — a PV surviving a Pod restart isn’t the same as surviving accidental deletion or corruption.
Common Mistakes
- Requesting
ReadWriteManyon a cloud block storage class that only supportsReadWriteOnce, leaving the PVC permanentlyPending. - Forgetting that
Deletereclaim policy destroys data the moment a PVC is removed — often discovered the hard way during a namespace cleanup. - Not setting resource requests correctly relative to actual usage, leading to volumes that fill up unexpectedly.
- Assuming PVC deletion also deletes the workload using it — it doesn’t prevent a Pod from remaining
Pendingafterward if it still references the (now missing) claim.
Volume Snapshots for Backup and Cloning
Beyond basic PV/PVC provisioning, the CSI storage framework supports VolumeSnapshot as a native Kubernetes object, giving you point-in-time backups of a volume without leaving the Kubernetes API:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: myapp-data-snapshot
spec:
volumeSnapshotClassName: csi-snapclass
source:
persistentVolumeClaimName: myapp-data
kubectl apply -f myapp-data-snapshot.yaml
kubectl get volumesnapshot myapp-data-snapshot
Restoring from a snapshot into a brand-new PVC (useful for cloning a production dataset into a staging environment, or recovering after data corruption) looks like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-data-restored
spec:
storageClassName: fast-ssd
dataSource:
name: myapp-data-snapshot
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
This requires your CSI driver to support the snapshot feature (most major cloud CSI drivers do) and the external-snapshotter controller installed in the cluster — worth confirming with kubectl get volumesnapshotclass before relying on it.
Local Persistent Volumes and When to Use Them
For latency-sensitive workloads where network-attached storage introduces unacceptable overhead — some distributed databases, certain caching layers — Kubernetes supports local PVs, backed by disk physically attached to a specific node:
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-pv-node1
spec:
capacity:
storage: 100Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: local-storage
local:
path: /mnt/disks/ssd1
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: ["node1"]
The tradeoff is significant: if that specific node is lost, the data is gone — there’s no automatic replication the way cloud block storage typically provides. Local volumes are appropriate specifically for workloads that handle their own replication at the application layer (many distributed databases do exactly this), not as a general-purpose storage solution.
Storage Performance Considerations
Different StorageClass configurations can have dramatically different performance characteristics, and it’s worth being deliberate rather than accepting whatever the cluster’s default happens to be:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: high-iops
provisioner: ebs.csi.aws.com
parameters:
type: io2
iopsPerGB: "50"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
For a database under real write load, the difference between a general-purpose gp3 volume and a provisioned-IOPS io2 volume can be the difference between consistent low-latency writes and periodic stalls under load — this is worth benchmarking with your actual workload rather than assuming defaults are adequate, especially once you’re past the prototyping stage.
Cleaning Up Orphaned Storage
Because Delete isn’t always the reclaim policy in use, and because StatefulSet deliberately leaves PVCs behind on scale-down, clusters accumulate unused PVCs and PVs over time if nobody’s watching. A periodic audit is worth scheduling:
kubectl get pvc -A -o json | jq -r '.items[] | select(.status.phase != "Bound") | "\(.metadata.namespace)/\(.metadata.name): \(.status.phase)"'
kubectl get pv -o json | jq -r '.items[] | select(.status.phase == "Released") | .metadata.name'
Released PVs (bound previously, but their PVC has since been deleted) sit around consuming real cloud storage cost until manually cleaned up or reclaimed — worth folding into a regular cost-review process rather than discovering a stack of orphaned volumes a year later.
Summary
Persistent storage in Kubernetes revolves around three cooperating objects: StorageClass defines what kinds of storage are available, PersistentVolumeClaim is how an application requests some, and PersistentVolume is the actual storage that gets bound to that request. For nearly all workloads, dynamic provisioning via a PVC referencing a StorageClass is the right approach — reach for static PV creation only when connecting to pre-existing external storage like NFS.