Loki, Grafana Labs’ log aggregation system, is often deployed as a multi-component StatefulSet-based system inside Kubernetes — with ingesters, distributors, queriers, and compactors all needing to stay available even while the cluster underneath them is being upgraded, drained, or autoscaled. If a node drain or cluster upgrade takes down too many Loki ingesters at once, you risk losing log data or breaking queries entirely. That’s exactly the problem Pod Disruption Budgets (PDBs) are built to solve. In this guide, I’ll explain what PDBs are and walk through configuring them specifically for a Loki deployment.
Voluntary vs. Involuntary Disruptions
Kubernetes distinguishes between two kinds of Pod disruption:
- Involuntary disruptions: Hardware failure, kernel panic, node running out of resources — things you can’t prevent, only recover from (via replication).
- Voluntary disruptions: Things initiated by an administrator or automation — draining a node for maintenance,
kubectl delete pod, cluster autoscaler scale-down, or a rolling node upgrade.
A PodDisruptionBudget only protects against voluntary disruptions. It tells Kubernetes: “No matter what maintenance operation you’re trying to perform, never take down more than X Pods (or leave fewer than Y available) from this set at once.”
Why This Matters for Loki Specifically
Loki’s architecture (especially in the microservices deployment mode) includes several stateful, quorum-sensitive components:
- Ingesters: Buffer and flush log data to long-term storage. Losing too many at once can cause data loss for logs not yet flushed.
- Distributors: Stateless, but if too many go down simultaneously, ingestion throughput craters.
- Queriers / Query Frontends: Handle read traffic; losing too many degrades query latency for everyone using Grafana dashboards.
- Compactor: Usually a single replica; losing it isn’t catastrophic short-term, but you don’t want it flapping constantly during node churn.
Because ingesters use a hash-ring based replication factor (commonly 3), losing more replicas than your replication factor tolerates during a rolling node upgrade can cause write failures or gaps in log ingestion. A PDB prevents the cluster from ever getting into that state during voluntary operations.
Kubernetes Architecture Refresher: Where PDBs Fit
PDBs are enforced by the Eviction API, which is what kubectl drain and the cluster autoscaler use instead of a raw delete. When something calls the Eviction API against a Pod:
- The API server checks if any PDB covers that Pod (via label selector).
- If evicting the Pod would violate the PDB’s
minAvailableormaxUnavailable, the eviction is rejected with a 429 Too Many Requests. - The caller (drain, autoscaler) retries later, once the Pod count is back in a safe state.
Note: a PDB does not stop kubectl delete pod directly — it only governs the Eviction API. This distinction trips people up often.
Step 1: Deploy Loki with Helm
Assuming you’re using the official loki Helm chart in microservices mode:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki -n logging --create-namespace \
--set loki.commonConfig.replication_factor=3 \
--set deploymentMode=Distributed
Confirm the components are running:
kubectl get pods -n logging
You should see distributor, ingester, querier, query-frontend, and compactor Pods.
Step 2: Create a PDB for Loki Ingesters
Ingesters are the most disruption-sensitive component. With a replication factor of 3, you want to guarantee at least 2 are always available:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: loki-ingester-pdb
namespace: logging
spec:
minAvailable: 2
selector:
matchLabels:
app.kubernetes.io/name: loki
app.kubernetes.io/component: ingester
Apply it:
kubectl apply -f loki-ingester-pdb.yaml
kubectl get pdb -n logging
Expected output:
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
loki-ingester-pdb 2 N/A 1 10s
The ALLOWED DISRUPTIONS column tells you exactly how many Pods can be evicted right now without violating the budget — this is the number you should watch during a maintenance window.
Step 3: Create PDBs for Other Components
For distributors and queriers (stateless, but you still want availability during rolling maintenance), maxUnavailable is often more practical than minAvailable since it scales naturally with replica count:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: loki-distributor-pdb
namespace: logging
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: loki
app.kubernetes.io/component: distributor
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: loki-querier-pdb
namespace: logging
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: loki
app.kubernetes.io/component: querier
Step 4: Configuring PDBs via the Helm Chart Values
Rather than managing PDBs as separate manifests, the Loki Helm chart supports PDB configuration natively — which is the cleaner, GitOps-friendly approach:
# values.yaml
ingester:
replicas: 3
podDisruptionBudget:
maxUnavailable: 1
querier:
replicas: 3
podDisruptionBudget:
maxUnavailable: 1
distributor:
replicas: 3
podDisruptionBudget:
maxUnavailable: 1
helm upgrade loki grafana/loki -n logging -f values.yaml
Step 5: Test It — Simulate a Node Drain
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
If the drain would violate a PDB, you’ll see output like:
error when evicting pods/"loki-ingester-1" -n "logging": Cannot evict pod as it would violate the pod's disruption budget.
Kubernetes will keep retrying automatically as Pods on other nodes become healthy again, until it’s safe to proceed.
Monitoring Allowed Disruptions
For ongoing visibility, especially before a planned maintenance window, script a quick check:
kubectl get pdb -n logging -o custom-columns=\
NAME:.metadata.name,MIN_AVAIL:.spec.minAvailable,MAX_UNAVAIL:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed
If ALLOWED is 0 for any Loki component, that’s a signal something is already degraded — investigate before starting maintenance, not after.
Common Mistakes
- Setting
minAvailableequal to total replica count — this makes the PDB impossible to satisfy during any voluntary disruption, effectively blocking node drains forever. - Using label selectors that are too broad, accidentally covering Pods from other apps and creating confusing eviction blocks.
- Forgetting that PDBs don’t protect against involuntary disruptions (a node crashing outright bypasses the Eviction API entirely).
- Not aligning
minAvailable/maxUnavailablewith the actual replication factor configured in Loki’scommonConfig.replication_factor— the PDB should reflect the same fault tolerance the application itself expects.
High Availability and Disaster Recovery Considerations
- Spread Loki ingester replicas across multiple availability zones using pod anti-affinity or topology spread constraints, so a single zone failure doesn’t take out your entire replication factor at once — PDBs and anti-affinity work together, not as substitutes for each other.
- Back up Loki’s storage backend (S3, GCS, or equivalent) independently; PDBs protect availability during maintenance, not data durability.
- Combine PDBs with readiness probes — a Pod that’s technically “Running” but failing readiness checks doesn’t count as available, which affects how the PDB math is evaluated.
Summary
Pod Disruption Budgets are a small but critical piece of running Loki reliably on Kubernetes. By setting sensible minAvailable/maxUnavailable values per component — aligned with Loki’s own replication factor — you ensure that routine cluster maintenance never accidentally causes log ingestion failures or query outages. Configure them through the Helm chart for consistency, verify them with kubectl get pdb, and always test a drain in a non-production cluster before you rely on this in an actual maintenance window.