How to Use Persistent Volume Claims in Kubernetes

Containers are ephemeral by design — kill a pod and everything written to its filesystem vanishes with it. That’s fine for stateless web servers, but the moment you need a database, a message queue, or any workload that has to survive a restart, you need durable storage. This is where PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) come in, and in this guide I’ll walk through the full storage model in Kubernetes, from the abstractions down to real EBS-backed manifests on AWS EKS.

The Storage Abstraction Model

Kubernetes storage has three layers that build on each other:

1. PersistentVolume (PV) — a piece of actual storage in the cluster, provisioned either manually by an admin or dynamically by a StorageClass. This is a cluster-scoped resource.
2. PersistentVolumeClaim (PVC) — a request for storage by a user/application, specifying size and access mode. This is namespace-scoped.
3. StorageClass — describes a “class” of storage (e.g., gp3 EBS volumes) and enables dynamic provisioning, so PVs get created automatically when a PVC references that class.

Think of it like this: a StorageClass is a template, a PV is an actual provisioned disk, and a PVC is a claim ticket a pod uses to get bound to a PV.

Access Modes

PVs support different access modes depending on the underlying storage backend:

• ReadWriteOnce (RWO) — mounted read-write by a single node (most common, e.g., EBS)
• ReadOnlyMany (ROX) — mounted read-only by many nodes
• ReadWriteMany (RWX) — mounted read-write by many nodes simultaneously (needs something like EFS, not EBS)
• ReadWriteOncePod (RWOP) — since Kubernetes 1.22, restricts mounting to a single pod rather than just a single node, useful for strict single-writer guarantees

StorageClasses on EKS

AWS EKS ships with the EBS CSI driver (as an add-on) providing dynamic provisioning. A typical production StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

kubectl apply -f gp3-storageclass.yaml
kubectl get storageclass

NAME            PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
gp3 (default)   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true

volumeBindingMode: WaitForFirstConsumer matters a lot on EKS — it delays provisioning until a pod is actually scheduled, so the EBS volume gets created in the same Availability Zone as the node the pod lands on. Using Immediate binding instead can create a volume in the wrong AZ, leaving the pod permanently Pending because EBS volumes can’t attach cross-AZ.

Creating a PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi

kubectl apply -f postgres-pvc.yaml
kubectl get pvc -n database

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-data   Pending  --                                          --         --             gp3            5s

Notice the PVC stays Pending until a pod actually consumes it — that’s WaitForFirstConsumer in action. It only binds once scheduled.

Using a PVC in a Pod

apiVersion: v1
kind: Pod
metadata:
  name: postgres
  namespace: database
spec:
  containers:
    - name: postgres
      image: postgres:16
      env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
      ports:
        - containerPort: 5432
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
          subPath: postgres
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: postgres-data

kubectl apply -f postgres-pod.yaml
kubectl get pvc -n database

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
postgres-data   Bound    pvc-3f2a1b8e-9c4d-4e21-a2f1-8d7c6b5a4e3f   20Gi       RWO            gp3

Note the subPath: postgres — this is a common pattern to avoid Postgres complaining about a non-empty lost+found directory that EBS volumes sometimes create at the root.

Statically Provisioned PVs (Manual)

Dynamic provisioning is the norm on EKS, but understanding static provisioning helps with troubleshooting and edge cases (like reattaching to a pre-existing EBS volume):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-manual
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: gp3
  csi:
    driver: ebs.csi.aws.com
    volumeHandle: vol-0abcd1234efgh5678
    fsType: ext4

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-manual
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi
  volumeName: postgres-pv-manual

Reclaim Policies

• Delete — the PV and underlying storage are deleted when the PVC is deleted. Default for dynamically provisioned volumes.
• Retain — the PV and its data survive PVC deletion, but the PV becomes Released and must be manually cleaned up or rebound before reuse.

For production databases, I strongly recommend switching critical StorageClasses to Retain — accidentally deleting a PVC (via a bad kubectl delete -f or a Helm uninstall) shouldn’t silently destroy your production data.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-retain
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Expanding a PVC

If allowVolumeExpansion: true is set on the StorageClass, you can grow a volume without recreating it:

kubectl patch pvc postgres-data -n database -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
kubectl get pvc postgres-data -n database

For most CSI drivers including EBS, the underlying volume expansion is online, but the filesystem resize inside the pod may require a pod restart depending on the filesystem and kubelet version:

kubectl rollout restart statefulset/postgres -n database

ReadWriteMany with EFS

For workloads needing shared access from multiple pods/nodes simultaneously — shared file uploads, ML training data, CMS media libraries — EBS won’t work since it’s RWO-only. Use the EFS CSI driver instead:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-0123456789abcdef0
  directoryPerms: "700"

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-uploads
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 100Gi

Multiple pods across multiple nodes can now mount shared-uploads concurrently.

PVCs in StatefulSets via volumeClaimTemplates

For workloads needing per-replica dedicated storage (databases, Kafka, Elasticsearch), StatefulSets use volumeClaimTemplates to automatically create one PVC per pod replica — covered in more depth in the companion StatefulSets article, but worth showing the storage angle here:

volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 20Gi

Each replica (postgres-0, postgres-1, …) gets its own PVC (data-postgres-0, data-postgres-1, …), and crucially, when a StatefulSet pod is rescheduled, it reattaches to the same PVC rather than getting a fresh empty volume.

Volume Snapshots for Backup and Restore

The EBS CSI driver supports the VolumeSnapshot API, giving you point-in-time backups without needing a separate backup tool for basic cases:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Retain

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-data-snapshot
  namespace: database
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: postgres-data

kubectl apply -f postgres-snapshot.yaml
kubectl get volumesnapshot -n database

Restoring from a snapshot into a new PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-restored
  namespace: database
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi
  dataSource:
    name: postgres-data-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

For production databases, pair this with a CronJob that snapshots on a schedule, and consider AWS Backup for centralized retention policies and cross-region copy, since raw VolumeSnapshot objects alone don’t give you automated scheduling or lifecycle management out of the box.

Cloning an Existing PVC

Beyond snapshot-based restore, CSI also supports directly cloning a live PVC — useful for spinning up a copy of production data in a staging environment for debugging:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data-clone
  namespace: staging
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi
  dataSource:
    name: postgres-data
    kind: PersistentVolumeClaim

Note the source and destination PVC must be in the same namespace for cloning (unlike snapshots, which can be restored into a different namespace) — if you need a cross-namespace copy, go through the snapshot-and-restore path instead.

Troubleshooting

PVC stuck Pending:

kubectl describe pvc postgres-data -n database

Look at the Events section — common causes are no matching StorageClass, WaitForFirstConsumer waiting on pod scheduling, or exceeding available capacity/quota.

Pod stuck Pending referencing a bound PVC:

kubectl describe pod postgres -n database

Often an AZ mismatch — the EBS volume was created in us-east-1a but the pod is only schedulable in us-east-1b. Check node labels:

kubectl get nodes -L topology.kubernetes.io/zone
kubectl get pv postgres-pv-manual -o jsonpath='{.metadata.labels}'

Volume mount errors after node replacement:

EBS volumes are AZ-locked and node-attached — if Cluster Autoscaler or Karpenter replaces a node, the CSI driver needs to detach and reattach the volume to the new node, which can take 30-60 seconds. If pods hang longer than that in ContainerCreating, check the EBS CSI controller pods:

kubectl logs -n kube-system -l app=ebs-csi-controller -c ebs-plugin --tail=50

Best Practices

• Always set volumeBindingMode: WaitForFirstConsumer on EKS to avoid AZ mismatch failures.
• Use Retain reclaim policy for anything holding production data you can’t afford to lose to an accidental PVC deletion.
• Enable allowVolumeExpansion: true on all StorageClasses by default — it costs nothing and saves painful volume migrations later.
• Set resource requests on pods using large PVCs realistically — EBS volume performance (IOPS/throughput) is tied to volume type and size, so undersized gp3 volumes with default IOPS can bottleneck database workloads.
• Snapshot regularly using the EBS CSI driver’s VolumeSnapshot support for backup/restore, not just relying on reclaim policy.

Common Mistakes

• Using ReadWriteMany access mode with a gp3/EBS StorageClass — this will simply fail to provision, since EBS doesn’t support RWX.
• Deleting a PVC without realizing the reclaim policy is Delete, and losing the underlying data permanently.
• Not accounting for the 30-60 second EBS attach/detach latency in pod disruption budgets or readiness probe timeouts during node rotations.
• Hardcoding a specific AZ’s node group without matching StorageClass topology constraints, leading to stranded volumes.

Summary

PersistentVolumeClaims are how Kubernetes workloads request durable storage without needing to know the underlying infrastructure details — that abstraction is handled by PersistentVolumes and StorageClasses, with dynamic provisioning tying them together automatically. On AWS EKS, this typically means the EBS CSI driver for RWO block storage (databases, single-writer workloads) and the EFS CSI driver for RWX shared file storage. Get WaitForFirstConsumer and reclaim policy right from the start — they’re the two settings most likely to cause painful production incidents if misconfigured.

References

• Kubernetes Documentation: Persistent Volumes
• Kubernetes Documentation: Storage Classes
• AWS EBS CSI Driver Documentation
• AWS EFS CSI Driver Documentation
• AWS EKS Best Practices Guide: Storage