How to Set Up Pod Disruption Budgets with Loki in Kubernetes

Loki, Grafana Labs’ log aggregation system, is often deployed as a multi-component StatefulSet-based system inside Kubernetes — with ingesters, distributors, queriers, and compactors all needing to stay available even while the cluster underneath them is being upgraded, drained, or autoscaled. If a node drain or cluster upgrade takes down too many Loki ingesters at once, you risk losing log data or breaking queries entirely. That’s exactly the problem Pod Disruption Budgets (PDBs) are built to solve. In this guide, I’ll explain what PDBs are and walk through configuring them specifically for a Loki deployment.

Voluntary vs. Involuntary Disruptions

Kubernetes distinguishes between two kinds of Pod disruption:

• Involuntary disruptions: Hardware failure, kernel panic, node running out of resources — things you can’t prevent, only recover from (via replication).
• Voluntary disruptions: Things initiated by an administrator or automation — draining a node for maintenance, kubectl delete pod, cluster autoscaler scale-down, or a rolling node upgrade.

A PodDisruptionBudget only protects against voluntary disruptions. It tells Kubernetes: “No matter what maintenance operation you’re trying to perform, never take down more than X Pods (or leave fewer than Y available) from this set at once.”

Why This Matters for Loki Specifically

Loki’s architecture (especially in the microservices deployment mode) includes several stateful, quorum-sensitive components:

• Ingesters: Buffer and flush log data to long-term storage. Losing too many at once can cause data loss for logs not yet flushed.
• Distributors: Stateless, but if too many go down simultaneously, ingestion throughput craters.
• Queriers / Query Frontends: Handle read traffic; losing too many degrades query latency for everyone using Grafana dashboards.
• Compactor: Usually a single replica; losing it isn’t catastrophic short-term, but you don’t want it flapping constantly during node churn.

Because ingesters use a hash-ring based replication factor (commonly 3), losing more replicas than your replication factor tolerates during a rolling node upgrade can cause write failures or gaps in log ingestion. A PDB prevents the cluster from ever getting into that state during voluntary operations.

Kubernetes Architecture Refresher: Where PDBs Fit

PDBs are enforced by the Eviction API, which is what kubectl drain and the cluster autoscaler use instead of a raw delete. When something calls the Eviction API against a Pod:

1. The API server checks if any PDB covers that Pod (via label selector).
2. If evicting the Pod would violate the PDB’s minAvailable or maxUnavailable, the eviction is rejected with a 429 Too Many Requests.
3. The caller (drain, autoscaler) retries later, once the Pod count is back in a safe state.

Note: a PDB does not stop kubectl delete pod directly — it only governs the Eviction API. This distinction trips people up often.

Step 1: Deploy Loki with Helm

Assuming you’re using the official loki Helm chart in microservices mode:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki -n logging --create-namespace \
  --set loki.commonConfig.replication_factor=3 \
  --set deploymentMode=Distributed

Confirm the components are running:

kubectl get pods -n logging

You should see distributor, ingester, querier, query-frontend, and compactor Pods.

Step 2: Create a PDB for Loki Ingesters

Ingesters are the most disruption-sensitive component. With a replication factor of 3, you want to guarantee at least 2 are always available:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-ingester-pdb
  namespace: logging
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: ingester

Apply it:

kubectl apply -f loki-ingester-pdb.yaml
kubectl get pdb -n logging

Expected output:

NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
loki-ingester-pdb  2               N/A                1                     10s

The ALLOWED DISRUPTIONS column tells you exactly how many Pods can be evicted right now without violating the budget — this is the number you should watch during a maintenance window.

Step 3: Create PDBs for Other Components

For distributors and queriers (stateless, but you still want availability during rolling maintenance), maxUnavailable is often more practical than minAvailable since it scales naturally with replica count:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-distributor-pdb
  namespace: logging
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: distributor
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-querier-pdb
  namespace: logging
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: querier

Step 4: Configuring PDBs via the Helm Chart Values

Rather than managing PDBs as separate manifests, the Loki Helm chart supports PDB configuration natively — which is the cleaner, GitOps-friendly approach:

# values.yaml
ingester:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

querier:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

distributor:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

helm upgrade loki grafana/loki -n logging -f values.yaml

Step 5: Test It — Simulate a Node Drain

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

If the drain would violate a PDB, you’ll see output like:

error when evicting pods/"loki-ingester-1" -n "logging": Cannot evict pod as it would violate the pod's disruption budget.

Kubernetes will keep retrying automatically as Pods on other nodes become healthy again, until it’s safe to proceed.

Monitoring Allowed Disruptions

For ongoing visibility, especially before a planned maintenance window, script a quick check:

kubectl get pdb -n logging -o custom-columns=\
NAME:.metadata.name,MIN_AVAIL:.spec.minAvailable,MAX_UNAVAIL:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

If ALLOWED is 0 for any Loki component, that’s a signal something is already degraded — investigate before starting maintenance, not after.

Common Mistakes

• Setting minAvailable equal to total replica count — this makes the PDB impossible to satisfy during any voluntary disruption, effectively blocking node drains forever.
• Using label selectors that are too broad, accidentally covering Pods from other apps and creating confusing eviction blocks.
• Forgetting that PDBs don’t protect against involuntary disruptions (a node crashing outright bypasses the Eviction API entirely).
• Not aligning minAvailable/maxUnavailable with the actual replication factor configured in Loki’s commonConfig.replication_factor — the PDB should reflect the same fault tolerance the application itself expects.

High Availability and Disaster Recovery Considerations

• Spread Loki ingester replicas across multiple availability zones using pod anti-affinity or topology spread constraints, so a single zone failure doesn’t take out your entire replication factor at once — PDBs and anti-affinity work together, not as substitutes for each other.
• Back up Loki’s storage backend (S3, GCS, or equivalent) independently; PDBs protect availability during maintenance, not data durability.
• Combine PDBs with readiness probes — a Pod that’s technically “Running” but failing readiness checks doesn’t count as available, which affects how the PDB math is evaluated.

Summary

Pod Disruption Budgets are a small but critical piece of running Loki reliably on Kubernetes. By setting sensible minAvailable/maxUnavailable values per component — aligned with Loki’s own replication factor — you ensure that routine cluster maintenance never accidentally causes log ingestion failures or query outages. Configure them through the Helm chart for consistency, verify them with kubectl get pdb, and always test a drain in a non-production cluster before you rely on this in an actual maintenance window.

References

• Kubernetes Documentation: Pod Disruption Budgets
• Grafana Loki Documentation
• Grafana Loki Helm Chart
• CNCF Loki Project Page