How to Set Up Pod Disruption Budgets with Prometheus in Kubernetes

If you’ve ever watched a node drain during a cluster upgrade and seen half your application’s pods disappear at once, you already understand why Pod Disruption Budgets (PDBs) exist. I’ve been burned by this before — a routine node pool upgrade took down enough replicas of a checkout service that we breached our SLA in about ninety seconds. That incident is the reason I now treat PDBs as a non-negotiable part of any production workload, and pairing them with Prometheus monitoring is what actually lets you trust that they’re working as intended.

In this article, I’ll walk through what PDBs are, how Kubernetes enforces them internally, how to write the YAML, and — critically — how to observe and alert on PDB behavior using Prometheus so you’re not flying blind during disruptions.

What Is a Pod Disruption Budget?

A PDB is a Kubernetes API object that tells the control plane how much voluntary disruption your application can tolerate. Voluntary disruptions are things initiated by cluster operators or automation — node drains, cluster autoscaler scale-downs, kubectl drain, or Cluster Autoscaler evictions. PDBs do not protect against involuntary disruptions like a node crashing or an OOM kill; there’s no way to budget for hardware failure.

A PDB specifies either:

• minAvailable — the minimum number (or percentage) of pods that must remain available
• maxUnavailable — the maximum number (or percentage) of pods that can be unavailable at once

The Eviction API respects these budgets. When something tries to evict a pod covered by a PDB, the API server checks whether the eviction would violate the budget. If it would, the eviction is rejected with a 429 Too Many Requests response, and the caller (usually kubectl drain or the node controller) retries later.

How PDBs Work Internally

Under the hood, the disruption controller — part of kube-controller-manager — continuously watches PDB objects and the pods they select via label selectors. It calculates:

• currentHealthy: pods currently passing readiness checks
• desiredHealthy: the target based on minAvailable/maxUnavailable
• disruptionsAllowed: how many more pods can be evicted right now without violating the budget

This state is written back into the PDB’s .status field, which is exactly what we’ll scrape with Prometheus later.

kubectl get pdb my-app-pdb -o yaml

status:
  currentHealthy: 4
  desiredHealthy: 3
  disruptionsAllowed: 1
  expectedPods: 4
  observedGeneration: 1

disruptionsAllowed: 1 means exactly one more pod can be evicted voluntarily before the disruption controller starts rejecting eviction requests.

Writing a Pod Disruption Budget

Here’s a basic PDB for a Deployment running 4 replicas, where I want at least 3 available at all times:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-service-pdb
  namespace: production
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: checkout-service

Alternatively, using maxUnavailable, which scales more gracefully if you change replica counts:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-service-pdb
  namespace: production
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout-service

I generally prefer maxUnavailable for stateless services since it scales automatically as you adjust replicas, and minAvailable for things where I care about an absolute floor, like a 3-node etcd-backed operator.

Apply it:

kubectl apply -f checkout-service-pdb.yaml
kubectl get pdb -n production

NAME                    MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
checkout-service-pdb    3               N/A               1                     12s

Common Mistakes with PDBs

A few things trip people up constantly:

• Setting minAvailable equal to replica count. This blocks all voluntary evictions, which will make node drains hang forever. I’ve seen this cause cluster upgrades to stall for hours.
• Selector mismatches. If your PDB’s matchLabels doesn’t align with your pod template labels, the PDB silently applies to zero pods and gives you false confidence.
• PDBs on single-replica Deployments. With minAvailable: 1 and only one replica, you’ve effectively blocked all voluntary disruption of that pod, which can interfere with node maintenance.
• Forgetting PDBs cover StatefulSets too. People often only apply them to Deployments, but StatefulSet pods being drained ungracefully is arguably worse.

Deploying Prometheus to Monitor PDBs

Now for the part that actually closes the loop: observability. Kubernetes exposes PDB status through kube-state-metrics, which Prometheus scrapes. If you don’t already have kube-state-metrics and Prometheus running, the fastest path is the kube-prometheus-stack Helm chart.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

Verify kube-state-metrics is running and exposing PDB metrics:

kubectl get pods -n monitoring | grep kube-state-metrics
kubectl port-forward -n monitoring svc/kube-prometheus-kube-state-metrics 8080:8080
curl localhost:8080/metrics | grep pod_disruption_budget

You should see metrics like:

kube_poddisruptionbudget_status_current_healthy{namespace="production",poddisruptionbudget="checkout-service-pdb"} 4
kube_poddisruptionbudget_status_desired_healthy{namespace="production",poddisruptionbudget="checkout-service-pdb"} 3
kube_poddisruptionbudget_status_expected_pods{namespace="production",poddisruptionbudget="checkout-service-pdb"} 4
kube_poddisruptionbudget_status_pod_disruptions_allowed{namespace="production",poddisruptionbudget="checkout-service-pdb"} 1

These four metrics are the entire foundation of PDB observability.

Writing PromQL Queries for PDB Health

To check which PDBs currently allow zero disruptions (meaning any additional voluntary eviction would be blocked):

kube_poddisruptionbudget_status_pod_disruptions_allowed == 0

To find PDBs where current healthy pods have dropped below desired:

kube_poddisruptionbudget_status_current_healthy
  < kube_poddisruptionbudget_status_desired_healthy

To calculate disruption budget utilization as a percentage:

(kube_poddisruptionbudget_status_expected_pods - kube_poddisruptionbudget_status_current_healthy)
  / kube_poddisruptionbudget_status_expected_pods * 100

Setting Up Alerting Rules

I define a PrometheusRule so that on-call gets paged before a PDB actually blocks a node drain, not after:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: pdb-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
    - name: pod-disruption-budgets
      rules:
        - alert: PDBNoDisruptionsAllowed
          expr: kube_poddisruptionbudget_status_pod_disruptions_allowed == 0
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "PDB {{ $labels.poddisruptionbudget }} allows zero disruptions"
            description: "Namespace {{ $labels.namespace }} PDB {{ $labels.poddisruptionbudget }} has 0 disruptions allowed for 10 minutes. Node drains may be blocked."

        - alert: PDBUnhealthy
          expr: kube_poddisruptionbudget_status_current_healthy < kube_poddisruptionbudget_status_desired_healthy
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "PDB {{ $labels.poddisruptionbudget }} is below desired healthy pods"
            description: "Current healthy pods are below the desired threshold, indicating a real availability problem."

Apply it:

kubectl apply -f pdb-alerts.yaml

Building a Grafana Dashboard

Since kube-prometheus-stack ships Grafana by default, I add a simple panel using the queries above — a table of poddisruptionbudget, disruptions_allowed, and current_healthy vs desired_healthy, plus a time series graph tracking disruptions_allowed over time so you can see it trend toward zero before an incident, not during one.

kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80

Default credentials are admin / the value in the kube-prometheus-grafana secret, retrievable with:

kubectl get secret -n monitoring kube-prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 -d

RBAC for PDB Management

Not everyone on a team should be able to create or modify PDBs — a poorly-set minAvailable can block a node drain for hours, and a maliciously low maxUnavailable could be used to prevent legitimate maintenance. I scope this with a dedicated Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pdb-editor
  namespace: production
rules:
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pdb-editor-binding
  namespace: production
subjects:
  - kind: Group
    name: platform-team
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pdb-editor
  apiGroup: rbac.authorization.k8s.io

Read-only access for developers who need visibility but shouldn’t modify budgets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pdb-viewer
  namespace: production
rules:
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["get", "list", "watch"]

Troubleshooting Common PDB and Prometheus Issues

PDB shows disruptionsAllowed: 0 unexpectedly. First check actual pod health — a PDB with minAvailable: 3 on 4 desired replicas will show zero allowed disruptions the moment even one pod is unready, not just when pods are missing entirely:

kubectl get pods -n production -l app=checkout-service
kubectl describe pdb checkout-service-pdb -n production

Look specifically at the Events section of the describe output — the disruption controller logs why it’s calculating the numbers it is.

kube-state-metrics not exposing PDB metrics at all. This usually means the resource isn’t being watched due to RBAC restrictions on kube-state-metrics’ own ClusterRole:

kubectl logs -n monitoring -l app.kubernetes.io/name=kube-state-metrics | grep -i poddisruptionbudget

If you see permission errors, the ClusterRole bundled with kube-state-metrics needs policy/poddisruptionbudgets in its get/list/watch verbs — this is included by default in recent chart versions but worth double-checking after upgrades.

Prometheus not scraping kube-state-metrics. Confirm the ServiceMonitor (if using the Prometheus Operator) actually targets the right service:

kubectl get servicemonitor -n monitoring
kubectl get endpoints -n monitoring kube-prometheus-kube-state-metrics

An empty endpoints list means the Service selector doesn’t match any running pods — usually a label mismatch introduced by a Helm values override.

Real-World Example: Rolling Node Upgrades

Here’s how this all comes together during an actual cluster upgrade. Before draining a node, I check the aggregate disruption budget health across the namespace:

kubectl get pdb -n production -o custom-columns=NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed,DESIRED:.status.desiredHealthy,CURRENT:.status.currentHealthy

If any PDB shows ALLOWED: 0, I hold off draining nodes hosting pods covered by that PDB until either replica count increases or traffic (and therefore desired health) naturally decreases. The Grafana dashboard built earlier makes this a 5-second visual check instead of manually running kubectl across every namespace.

kubectl drain node-i-0abc123 --ignore-daemonsets --delete-emptydir-data --timeout=300s

If this command hangs, it’s almost always a PDB blocking the final eviction — kubectl drain will print exactly which pod and PDB is holding things up, and the Prometheus alert defined earlier should have already paged someone before it got to this point.

Production Considerations and CI/CD Integration

In real deployments, I bake PDB manifests directly into the same Helm chart or Kustomize overlay as the Deployment itself, so they’re never deployed out of sync. A CI pipeline step I use looks like this:

# .github/workflows/deploy.yml (excerpt)
- name: Validate PDB coverage
  run: |
    kubectl apply -f k8s/pdb.yaml --dry-run=server
    kubectl get deployment checkout-service -o jsonpath='{.spec.replicas}'

I also run a small policy check (via OPA/Gatekeeper or Kyverno) that rejects any Deployment with more than 1 replica that lacks a corresponding PDB — this has saved me from repeating the incident I mentioned at the start.

High Availability and Disaster Recovery Notes

• Always pair PDBs with anti-affinity rules (podAntiAffinity) so pods aren’t concentrated on one node in the first place — a PDB doesn’t help if all your “available” pods are on the node being drained.
• During a real cluster disaster (not a voluntary drain), PDBs offer no protection — that’s what multi-AZ deployments and readiness/liveness probes are for.
• Test PDB behavior deliberately with kubectl drain <node> --ignore-daemonsets in a staging cluster before you need it in production.

Summary

Pod Disruption Budgets are a small object with an outsized impact on availability during routine cluster operations. Writing the YAML takes two minutes; the real engineering work is verifying they behave correctly through Prometheus metrics like kube_poddisruptionbudget_status_pod_disruptions_allowed, alerting before disruptions are blocked, and visualizing PDB health in Grafana so you catch problems before an upgrade — not during one.

References

• Kubernetes Docs: Disruptions
• Kubernetes Docs: Specifying a Disruption Budget
• kube-state-metrics Documentation
• Prometheus Community Helm Charts
• CNCF Prometheus Project