Static Pod Disruption Budgets are fine until your availability requirements aren’t static. A payments service might tolerate losing 2 of 10 pods on a quiet Tuesday morning but need every single replica during a Black Friday traffic spike. That’s the gap I kept running into with plain PDBs — they don’t know about traffic, queue depth, or request latency, because those aren’t things the disruption controller has any visibility into. The workaround is combining PDBs with custom metrics, usually driving replica count (and therefore effective disruption tolerance) dynamically through the Horizontal Pod Autoscaler.
Why “PDBs with Custom Metrics” Isn’t Literal
It’s worth being precise here: PDB objects themselves only accept minAvailable/maxUnavailable as static integers or percentages — there’s no native field for wiring a PDB directly to a Prometheus query. What you’re actually building is a system where:
- Custom metrics (via Prometheus Adapter) drive a Horizontal Pod Autoscaler
- The HPA changes replica count based on real load
- A percentage-based PDB (
maxUnavailable: 25%, for example) automatically scales its effective protection alongside replica count - Optionally, a controller adjusts the PDB itself in response to metrics for cases percentage-based budgets can’t express
This combination is what people mean in practice when they say “PDBs with custom metrics.”
Architecture Overview
Prometheus ──scrapes──▶ App metrics (queue depth, latency, custom SLI)
│
▼
Prometheus Adapter (custom.metrics.k8s.io API)
│
▼
HorizontalPodAutoscaler ──scales──▶ Deployment replicas
│
▼
PodDisruptionBudget (percentage-based) ──protects──▶ scaled replica set
Step 1: Deploy Prometheus and the Prometheus Adapter
Assuming Prometheus is already running (via kube-prometheus-stack, as in earlier setups), install the adapter that exposes Prometheus queries through the Kubernetes custom metrics API:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-adapter prometheus-community/prometheus-adapter \
--namespace monitoring \
--set prometheus.url=http://kube-prometheus-kube-prome-prometheus.monitoring.svc \
--set prometheus.port=9090
Verify the custom metrics API is being served:
kubectl get apiservice v1beta1.custom.metrics.k8s.io
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
Step 2: Define a Custom Metrics Rule
Configure the adapter to expose a specific application metric — say, requests-per-second per pod — via a ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-adapter-config
namespace: monitoring
data:
config.yaml: |
rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
resources:
overrides:
namespace: {resource: "namespace"}
pod: {resource: "pod"}
name:
matches: "^http_requests_total"
as: "http_requests_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'
kubectl apply -f prometheus-adapter-config.yaml
kubectl rollout restart deployment prometheus-adapter -n monitoring
Confirm the metric is queryable:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second" | jq .
Step 3: Create the HPA Using the Custom Metric
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: checkout-service-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: checkout-service
minReplicas: 4
maxReplicas: 30
metrics:
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: "50"
kubectl apply -f hpa.yaml
kubectl get hpa -n production
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
checkout-service-hpa Deployment/checkout-service 42/50 4 30 6
Step 4: Percentage-Based PDB That Scales With the HPA
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-service-pdb
namespace: production
spec:
maxUnavailable: 25%
selector:
matchLabels:
app: checkout-service
With maxUnavailable: 25%, whether the HPA has scaled to 4 replicas or 30, the disruption tolerance automatically recalculates: at 4 replicas, 1 pod can go; at 30, 7 can go. This is the key insight — percentage-based PDBs are inherently “metric-aware” as long as something (the HPA, driven by custom metrics) is adjusting replica count in response to load.
Step 5: Handling Cases Percentages Can’t Express
Sometimes you need stricter logic — e.g., “never allow disruption if queue depth exceeds 10,000 messages,” regardless of replica count. This requires a small custom controller (or a scheduled Job) that patches the PDB directly based on a Prometheus query result.
A simple version using a CronJob and curl against Prometheus’s HTTP API:
apiVersion: batch/v1
kind: CronJob
metadata:
name: pdb-adjuster
namespace: production
spec:
schedule: "*/2 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: pdb-adjuster
containers:
- name: adjuster
image: curlimages/curl:8.7.1
command:
- /bin/sh
- -c
- |
QUEUE_DEPTH=$(curl -s "http://kube-prometheus-kube-prome-prometheus.monitoring.svc:9090/api/v1/query?query=queue_depth" | jq -r '.data.result[0].value[1]')
if [ "$(echo "$QUEUE_DEPTH > 10000" | bc)" -eq 1 ]; then
kubectl patch pdb checkout-service-pdb -n production --type=merge -p '{"spec":{"maxUnavailable":0}}'
else
kubectl patch pdb checkout-service-pdb -n production --type=merge -p '{"spec":{"maxUnavailable":"25%"}}'
fi
restartPolicy: OnFailure
RBAC for this job needs patch on poddisruptionbudgets scoped tightly to the namespace and resource name where possible.
apiVersion: v1
kind: ServiceAccount
metadata:
name: pdb-adjuster
namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pdb-patcher
namespace: production
rules:
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pdb-patcher-binding
namespace: production
subjects:
- kind: ServiceAccount
name: pdb-adjuster
namespace: production
roleRef:
kind: Role
name: pdb-patcher
apiGroup: rbac.authorization.k8s.io
Note: maxUnavailable: 0 on a PDB blocks all voluntary evictions — use this sparingly and only for genuinely critical windows, since it can also block node drains and cluster upgrades from proceeding.
Monitoring the Whole System
Track these together in Grafana:
http_requests_per_second(the driving metric)- HPA
desiredReplicasvscurrentReplicas kube_poddisruptionbudget_status_pod_disruptions_allowed
Seeing all three on one dashboard makes it obvious when your scaling and disruption protection are working in concert versus fighting each other.
Alternative Approach: KEDA for Metric-Driven Scaling
Everything above uses Prometheus Adapter and the native HPA, which is the most portable approach. If you’re already using or open to KEDA (Kubernetes Event-Driven Autoscaling), it offers a more ergonomic API for defining scaling triggers directly against Prometheus without hand-writing adapter configuration:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: checkout-service-scaledobject
namespace: production
spec:
scaleTargetRef:
name: checkout-service
minReplicaCount: 4
maxReplicaCount: 30
triggers:
- type: prometheus
metadata:
serverAddress: http://kube-prometheus-kube-prome-prometheus.monitoring.svc:9090
metricName: http_requests_per_second
query: sum(rate(http_requests_total{namespace="production"}[2m]))
threshold: "50"
helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace
kubectl apply -f scaledobject.yaml
KEDA creates and manages an HPA object under the hood, so the PDB interaction described throughout this article still applies unchanged — KEDA is really just a friendlier interface over the same underlying mechanism, with the added benefit of built-in scale-to-zero support that plain HPA doesn’t offer.
Troubleshooting Metric-Driven Scaling and PDB Interaction
HPA not scaling despite metric clearly exceeding threshold. Check whether the custom metrics API is actually returning fresh data:
kubectl get hpa checkout-service-hpa -n production -o yaml | grep -A 10 status
kubectl describe hpa checkout-service-hpa -n production
Look for FailedGetPodsMetric or unable to get metric events — this usually traces back to a Prometheus Adapter misconfiguration or the metric simply not existing for the queried time window.
PDB blocking a node drain during a scale-down event. This is actually expected behavior working correctly — if maxUnavailable: 25% and the HPA just scaled down aggressively, the PDB is doing its job by refusing to let node drains compound the reduction further. Check current allowed disruptions before assuming something’s broken:
kubectl get pdb checkout-service-pdb -n production -o jsonpath='{.status.disruptionsAllowed}'
PDB-adjuster CronJob left maxUnavailable: 0 stuck permanently. Add a dead-man’s-switch check — a separate scheduled job that alerts if the PDB has been at maxUnavailable: 0 for longer than your maximum acceptable window, independent of the adjuster’s own logic:
kube_poddisruptionbudget_status_pod_disruptions_allowed{poddisruptionbudget="checkout-service-pdb"} == 0
with a for: 30m alert threshold, since a legitimate high-queue-depth window should resolve well within that time in a healthy system.
Production Considerations
- Give the HPA a
stabilizationWindowSecondsin its scaling behavior to avoid flapping replica counts, which would cause the PDB’s effective protection to jitter too. - Avoid combining
maxUnavailable: 0with cluster-autoscaler-driven node scale-down — this can leave “stuck” nodes that never drain, inflating cost. - Always test the PDB-adjuster CronJob logic in staging with synthetic load before trusting it in production; a bug that pins
maxUnavailable: 0permanently is a self-inflicted incident.
Combining with Cluster Autoscaler Awareness
One subtlety worth calling out explicitly: when the HPA scales up in response to a custom metric, those new pods need somewhere to run. If the cluster is at capacity, the Cluster Autoscaler needs to provision new nodes first, which introduces a delay between “metric crosses threshold” and “PDB’s effective protection actually increases.” During that window, your disruption tolerance is still calculated against the old, smaller replica count — worth factoring into any incident response runbook that assumes PDBs update instantaneously alongside metrics.
kubectl get hpa checkout-service-hpa -n production -o jsonpath='{.status.desiredReplicas}'
kubectl get deployment checkout-service -n production -o jsonpath='{.status.replicas}'
A gap between desiredReplicas and actual running replicas is your signal that the PDB hasn’t “caught up” yet to the metric-driven scaling decision.
Common Mistakes
- Expecting the PDB API itself to accept a metrics query — it doesn’t; the dynamism has to come from replica count changes or an external patching mechanism.
- Setting HPA
minReplicaslower than what your PDB’s absolute floor (if usingminAvailable) requires — this creates a permanently blocked disruption state. - Not securing the RBAC on any custom controller that patches PDBs — this is a sensitive permission.
Summary
True custom-metric-aware PDBs aren’t a single Kubernetes object — they’re a small system built from Prometheus Adapter, HPA, percentage-based PDBs, and optionally a lightweight controller for edge cases. Most of the time, letting a percentage-based PDB ride along with HPA-driven scaling gets you 90% of the value with none of the custom controller complexity. Reach for the patching approach only when you have a genuine business rule that percentages can’t express.
