If you’re running workloads on Kubernetes, whether it’s a single cluster or a fleet of them across clouds, security can’t be an afterthought. This guide walks through the practical, battle-tested best practices I rely on when hardening clusters — from the control plane down to individual pods.

Why Kubernetes Security Deserves Special Attention

Kubernetes is powerful because it abstracts away infrastructure, but that same abstraction is what makes security tricky. You’re not just securing servers anymore — you’re securing an API, a scheduler, a networking layer, secret storage, and dozens of moving components that all trust each other by default.

A single overly permissive RBAC role, an exposed API server, or a container running as root can be the difference between a contained incident and a full cluster compromise.

The Kubernetes Security Model at a Glance

Before diving into specific practices, it helps to picture where the risk actually lives in a cluster.

flowchart TD
    A[Internet] -->|Ingress| B[API Server]
    B --> C[Authentication & RBAC]
    C --> D[Scheduler]
    C --> E[Controller Manager]
    D --> F[Nodes / Kubelet]
    F --> G[Pods & Containers]
    G --> H[Secrets & ConfigMaps]
    G --> I[Network Policies]
    style B fill:#f96,stroke:#333
    style G fill:#69f,stroke:#333

Every layer in this diagram is a place where a misconfiguration can quietly become an incident.

1. Harden the Kubernetes API Server

The API server is the front door to your entire cluster. If it’s compromised, everything behind it is at risk.

• Disable anonymous authentication (--anonymous-auth=false).
• Use strong authentication methods (OIDC, client certificates) instead of static tokens.
• Enable audit logging so you can trace who did what and when.
• Restrict access to the API server using network policies and firewall rules — it should never be reachable from the open internet without strict controls.

If you want a deeper walkthrough of this specific layer, I’ve written a dedicated Kubernetes API Server Hardening Guide that goes step by step through admission controllers and audit policies.

2. Apply the Principle of Least Privilege with RBAC

Role-Based Access Control is one of the most misused features in Kubernetes — not because it’s complicated, but because teams default to cluster-admin for convenience.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

Bind roles narrowly, scope them to namespaces, and audit them regularly. I go into much more depth on this in my Kubernetes RBAC Security Guide, including how to use ServiceAccounts safely, which I also cover separately when managing ServiceAccounts in Kubernetes.

3. Secure Secrets Properly

By default, Kubernetes Secrets are only base64-encoded, not encrypted. That’s not security — it’s obfuscation. Enable encryption at rest, use an external secrets manager (Vault, AWS Secrets Manager, or similar) where possible, and never bake secrets into container images.

I cover this topic thoroughly in How to Secure Kubernetes Secrets, including how Helm-based deployments should handle sensitive values, which is also discussed in my post on using Kubernetes Secrets with Helm.

4. Isolate Workloads with Network Policies

By default, every pod in a Kubernetes cluster can talk to every other pod. That’s a flat network, and it’s a gift to any attacker who lands a foothold. Network Policies let you define exactly which pods can talk to which.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Start with a default-deny policy, then open only the traffic paths your applications actually need. I break this down further in Kubernetes Network Policies Explained.

5. Enforce Pod Security Standards

Pods running as root, with privileged mode enabled, or with host path mounts are common culprits in container breakouts. Kubernetes’ built-in Pod Security Admission (replacing the older PodSecurityPolicy) lets you enforce baseline or restricted profiles cluster-wide.

• Run containers as non-root users.
• Set readOnlyRootFilesystem: true where possible.
• Drop unnecessary Linux capabilities.
• Avoid hostNetwork, hostPID, and hostIPC unless absolutely required.

I still reference my own notes on setting up Pod Security Policies in Kubernetes whenever I need a refresher on the admission controller flags.

6. Scan Images and Watch Runtime Behavior

Security doesn’t stop at deployment. You need to know what’s inside your images before they ship, and what’s happening inside your containers while they run. I’ve written separately about Container Image Scanning and the difference between that and Runtime Container Security — both are essential, and neither replaces the other.

7. Monitor, Audit, and Alert

Visibility is what turns “we got breached and didn’t know for three months” into “we caught it in ten minutes.” Tools like Prometheus for metrics and alerting are worth setting up early — I walk through this in Setting Up Prometheus Alerting in Kubernetes.

Common Kubernetes Security Mistakes to Avoid

• Using the default namespace for production workloads. Namespaces are cheap; use them to isolate blast radius.
• Leaving the Kubernetes Dashboard exposed without authentication.
• Skipping resource limits, which can lead to noisy-neighbor denial-of-service conditions.
• Not rotating certificates and tokens on a schedule.
• Trusting default service account tokens mounted automatically into every pod — disable auto-mounting where it isn’t needed.

Best Practices Checklist

1. Enable RBAC and review bindings quarterly.
2. Rotate and encrypt secrets; avoid plaintext in manifests.
3. Apply default-deny network policies per namespace.
4. Enforce restricted Pod Security Standards.
5. Scan images in CI/CD before deployment.
6. Enable audit logging and forward logs to a SIEM.
7. Keep Kubernetes and node OS versions patched.
8. Use admission controllers (OPA/Gatekeeper, Kyverno) to enforce policy as code.

FAQs

Is Kubernetes secure by default? No. Kubernetes ships with sensible defaults for functionality, not maximum security. You have to actively configure RBAC, network policies, and pod security settings.

What’s the single highest-impact change I can make today? Locking down RBAC and removing unnecessary cluster-admin bindings usually has the biggest immediate impact, followed closely by applying default-deny network policies.

Do I need a service mesh for security? Not strictly, but a service mesh like Istio or Linkerd adds mutual TLS between services and finer-grained traffic policy, which is valuable at scale.

How often should I audit my cluster’s security posture? At minimum quarterly, though continuous scanning with policy-as-code tools is far more effective than periodic manual reviews.

Conclusion

Kubernetes security isn’t a single setting you flip — it’s a layered discipline that spans the API server, RBAC, networking, secrets, workloads, and monitoring. Start with the fundamentals: least-privilege RBAC, default-deny networking, and non-root containers. From there, build out image scanning, runtime detection, and continuous auditing. None of this is glamorous work, but it’s the difference between a resilient cluster and a headline.

The post Kubernetes Security Best Practices: A Practical Guide for 2026 appeared first on Abdul Wahab Junaid.

In this guide, I’ll walk through what a custom controller actually is, how it fits into Kubernetes’ architecture, and how to build and deploy one from scratch — from the basic reconciliation loop all the way to production-grade concerns like leader election, RBAC, and observability.

What Is a Controller, Really?

At its core, Kubernetes is a declarative system built around a single idea: desired state versus actual state. You tell the API server what you want (via a manifest), and controllers continuously work to make the actual state match that desired state. This is called a reconciliation loop, and it’s the heartbeat of everything Kubernetes does.

A controller:

1. Watches the API server for changes to a resource (via watch on the API, backed by etcd).
2. Compares the observed state to the desired state.
3. Acts to reconcile the difference — creating, updating, or deleting resources as needed.
4. Repeats, forever, reacting to new events and periodically re-syncing.

Built-in controllers do this for native resources like Pods and Deployments. A custom controller does the exact same thing, but for resources you define yourself — usually a Custom Resource Definition (CRD) — or even for built-in resources, if you want to add your own behavior on top of what Kubernetes already does.

Kubernetes Architecture Refresher

Before writing a controller, it helps to understand where it lives in the broader architecture:

• API Server: The front door. All reads and writes go through it, and it persists objects into etcd.
• etcd: The cluster’s source of truth, a distributed key-value store.
• Scheduler: Assigns Pods to Nodes.
• kubelet: Runs on every node, actually starts/stops containers.
• Controller Manager: Runs the built-in controllers as control loops.
• Custom Controller: Just another client of the API server. It doesn’t need special privileges to “be” a controller — it authenticates like any other client, watches resources it cares about, and issues API calls to reconcile them.

This last point is important: a custom controller is not a special kind of process. It’s an ordinary program (often a Go binary, though not exclusively) that talks to the Kubernetes API using a client library, typically running as a Deployment inside the cluster it manages — or even outside the cluster, pointed at a kubeconfig.

When You Need a Custom Controller vs. a CRD Alone

A CRD by itself just defines a new object type — it’s schema and storage, nothing more. If you create a CRD for Website and apply a Website object, absolutely nothing happens unless something is watching for it. That “something” is the controller. Together, a CRD plus a controller form what’s commonly called an Operator — a controller with domain-specific knowledge that automates an operational task a human would otherwise do by hand.

Use a custom controller when:

• You’re building an Operator for a stateful application (databases, message queues, certificate authorities).
• You want to enforce custom policies (e.g., automatically injecting sidecars, labeling resources, or denying non-compliant objects).
• You need to bridge Kubernetes to an external system (cloud resources, DNS records, ticketing systems).

Building a Custom Controller: The Concepts

Most production controllers today are built with client-go, Kubernetes’ official Go client library, often through higher-level tooling like controller-runtime (used by the Operator SDK and Kubebuilder). The core building blocks are:

• Informer: Maintains a local, cache-synced copy of the resources you care about, so you’re not hammering the API server with reads.
• Lister: A read-only, cache-backed accessor built on top of the informer.
• Workqueue: A rate-limited queue that decouples “an event happened” from “process the event,” with automatic retries on failure.
• Reconciler: The function containing your actual business logic.

The typical flow looks like this:

API Server --watch--> Informer --enqueue--> Workqueue --dequeue--> Reconcile()

Step 1: Define a Custom Resource Definition

Let’s build a small controller that watches a custom Website resource and creates a Deployment and Service for it automatically.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: websites.example.com
spec:
  group: example.com
  names:
    kind: Website
    listKind: WebsiteList
    plural: websites
    singular: website
    shortNames: ["ws"]
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
              required: ["image"]
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
      subresources:
        status: {}

Apply it:

kubectl apply -f website-crd.yaml
kubectl get crd websites.example.com

Step 2: Scaffold the Controller with Kubebuilder

Kubebuilder gives you a working project skeleton in minutes:

kubebuilder init --domain example.com --repo github.com/yourname/website-operator
kubebuilder create api --group web --version v1 --kind Website

This generates a WebsiteReconciler struct with a Reconcile method — this is where your logic goes.

func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var website webv1.Website
    if err := r.Get(ctx, req.NamespacedName, &website); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      website.Name,
            Namespace: website.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &website.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": website.Name},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": website.Name},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "web",
                        Image: website.Spec.Image,
                    }},
                },
            },
        },
    }

    if err := ctrl.SetControllerReference(&website, deployment, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }

    if err := r.Create(ctx, deployment); err != nil && !apierrors.IsAlreadyExists(err) {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Notice the call to SetControllerReference — this sets an owner reference so that if the Website object is deleted, Kubernetes’ built-in garbage collector cleans up the Deployment automatically.

Step 3: RBAC for the Controller

A controller running inside the cluster needs a ServiceAccount and permissions scoped to exactly what it touches — nothing more.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: website-controller-role
rules:
  - apiGroups: ["web.example.com"]
    resources: ["websites", "websites/status"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: website-controller-binding
subjects:
  - kind: ServiceAccount
    name: website-controller
    namespace: website-system
roleRef:
  kind: ClusterRole
  name: website-controller-role
  apiGroup: rbac.authorization.k8s.io

Step 4: Deploy the Controller

Build and push the image, then deploy it as a standard Kubernetes Deployment:

make docker-build docker-push IMG=yourrepo/website-controller:v0.1.0
make deploy IMG=yourrepo/website-controller:v0.1.0
kubectl get pods -n website-system

Test it:

kubectl apply -f - <<EOF
apiVersion: web.example.com/v1
kind: Website
metadata:
  name: demo-site
spec:
  image: nginx:1.27
  replicas: 2
EOF

kubectl get deployment demo-site

You should see a Deployment named demo-site with 2 replicas appear automatically — created entirely by your controller.

Leader Election for High Availability

Running a single replica of your controller is a single point of failure. Run multiple replicas with leader election enabled so only one is actively reconciling at a time, with automatic failover:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "website-controller-leader",
})

The non-leader replicas sit idle, watching a Lease object, ready to take over instantly if the leader’s pod dies.

Observability and Troubleshooting

• Structured logging: Log reconcile events with the resource’s namespace/name for traceability.
• Metrics: controller-runtime exposes Prometheus metrics by default (workqueue_depth, reconcile_errors_total) on :8080/metrics.
• Events: Emit Kubernetes Events (kubectl describe website demo-site) so operators see what your controller is doing without digging through logs.
• Common mistakes: forgetting to handle NotFound errors after deletion, not setting owner references (leading to orphaned resources), and reconciling too aggressively without rate limiting, which can overwhelm the API server.

Production Best Practices

• Keep reconcile functions idempotent — they may be called repeatedly for the same state.
• Use finalizers if your controller needs to clean up external resources (cloud load balancers, DNS records) before an object is deleted.
• Version your CRD schema carefully; use conversion webhooks if you need to evolve the API.
• Set resource requests/limits on the controller Pod itself — a runaway controller can destabilize a cluster.
• For disaster recovery, remember that CRs are stored in etcd like everything else — back up etcd regularly if your CRDs hold important state.

CI/CD Integration

In a real DevOps pipeline, the controller image is built and tested in CI (unit tests with envtest, which spins up a real API server without a full cluster), then the manifests are deployed via GitOps tools like Argo CD or Flux, so any change to the CRD or controller Deployment is applied automatically and auditable through Git history.

Summary

Custom controllers are how Kubernetes becomes a true automation platform rather than just a container scheduler. By combining a CRD with a reconciliation loop, you can teach Kubernetes to manage anything — databases, certificates, cloud infrastructure — using the same declarative model it already uses for Pods and Services. Start small, keep your reconcile logic idempotent, lock down RBAC tightly, and add leader election and observability before you trust a controller with production traffic.

References

• Kubernetes Documentation: Custom Resources
• Kubebuilder Book
• client-go on GitHub
• CNCF Operator White Paper
• Kubernetes controller-runtime

The post How to Set Up Custom Controllers in Kubernetes appeared first on Abdul Wahab Junaid.

Why RBAC Matters

Kubernetes clusters often host multiple teams, multiple environments, and multiple layers of automation (CI/CD pipelines, controllers, monitoring agents) all talking to the same API server. Without access control, any authenticated user or service account could do anything — read secrets, delete deployments, modify RBAC itself. RBAC solves this by letting you define who can do what, on which resources, in which namespaces.

Kubernetes Architecture Context

Every request to Kubernetes — whether from kubectl, a controller, or a CI pipeline — goes through three stages at the API server:

1. Authentication: Who are you? (certificates, tokens, OIDC, service accounts)
2. Authorization: Are you allowed to do this? (this is where RBAC lives)
3. Admission Control: Should this specific request be allowed/modified? (webhooks, policies)

RBAC is the authorization layer. It doesn’t care who you are beyond your identity (user or service account) and group memberships — it just checks whether a Role permits the verb and resource you’re requesting.

The Four RBAC Objects

RBAC in Kubernetes is built from four API objects:

• Role: A set of permissions (verbs on resources) scoped to a single namespace.
• ClusterRole: The same thing, but scoped cluster-wide (or reusable across namespaces).
• RoleBinding: Grants a Role to a user, group, or service account within a namespace.
• ClusterRoleBinding: Grants a ClusterRole cluster-wide.

The key mental model: Roles/ClusterRoles define permissions; Bindings assign those permissions to someone. A Role by itself does nothing until it’s bound.

Step 1: Create a Namespace-Scoped Role

Let’s say a developer needs to view and manage Pods in the staging namespace, but nothing else.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: pod-manager
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Apply it:

kubectl apply -f pod-manager-role.yaml

Step 2: Bind the Role to a User

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-manager-binding
  namespace: staging
subjects:
  - kind: User
    name: jane.doe
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-manager
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f pod-manager-binding.yaml

Now Jane can manage Pods only inside staging — nothing else, and nowhere else.

Step 3: Cluster-Wide Permissions with ClusterRole

Suppose you have a monitoring service account that needs to read node metrics across the whole cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
  - apiGroups: [""]
    resources: ["nodes", "nodes/metrics"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-reader-binding
subjects:
  - kind: ServiceAccount
    name: monitoring-agent
    namespace: monitoring
roleRef:
  kind: ClusterRole
  name: node-reader
  apiGroup: rbac.authorization.k8s.io

Understanding Verbs and Resources

Common verbs: get, list, watch, create, update, patch, delete, deletecollection. Resources can be scoped further using resourceNames to restrict access to specific named objects:

rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["app-config"]
    verbs: ["get", "update"]

This grants access only to the app-config ConfigMap — nothing else in that resource type.

Aggregated ClusterRoles

For larger clusters, Kubernetes supports aggregated ClusterRoles, which combine multiple ClusterRoles using label selectors. This is how the built-in admin, edit, and view roles work internally, and it’s a clean pattern for building modular permission sets:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-aggregate
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules: []
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-endpoints
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules:
  - apiGroups: [""]
    resources: ["endpoints", "services", "pods"]
    verbs: ["get", "list", "watch"]

Checking Permissions

Before you go debugging why something isn’t working, use kubectl auth can-i:

kubectl auth can-i delete pods --namespace staging --as jane.doe
# yes

kubectl auth can-i create deployments --namespace production --as jane.doe
# no

This is invaluable for both testing and troubleshooting RBAC issues without waiting for a user to hit an error.

RBAC for Service Accounts (CI/CD and Automation)

Every Pod runs with a ServiceAccount, and by default it’s the default service account in its namespace, which typically has minimal permissions. For a CI/CD pipeline deploying to a namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployer
  namespace: production
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: ci-deployer
    namespace: production
roleRef:
  kind: Role
  name: deployer
  apiGroup: rbac.authorization.k8s.io

Your pipeline (GitHub Actions, GitLab CI, Argo CD) then authenticates using this ServiceAccount’s token, scoped to exactly the deployments it needs to touch.

Security Best Practices

• Principle of least privilege: Never grant cluster-admin unless absolutely necessary. Audit who has it with kubectl get clusterrolebindings -o json | jq.
• Avoid wildcard rules (resources: ["*"], verbs: ["*"]) in production; they’re a common source of privilege escalation.
• Use groups, not individual users, for bindings when integrating with an identity provider (OIDC) — it scales much better as teams grow.
• Separate namespaces per environment/team and scope Roles accordingly, rather than relying on a handful of broad ClusterRoles.
• Rotate and audit service account tokens, especially long-lived ones; prefer projected, time-bound tokens (BoundServiceAccountToken) which are the default in modern Kubernetes.
• Watch for privilege escalation via RBAC itself — a user with create on rolebindings and bind verb capability could grant themselves broader access; Kubernetes has built-in escalation checks, but review carefully.

Common Mistakes

• Binding a ClusterRole via a RoleBinding to try to limit it to a namespace — this actually works and is a valid pattern, but it’s easy to confuse with a ClusterRoleBinding, which grants access everywhere.
• Forgetting pods/log and pods/exec are separate subresources from pods — granting get on pods doesn’t let someone kubectl exec into them.
• Applying RBAC changes without testing with kubectl auth can-i --as, leading to either broken pipelines or unnoticed over-permissioning.

Troubleshooting RBAC Issues

When a request is denied, the API server returns a clear Forbidden error naming the exact verb, resource, and namespace it evaluated:

Error from server (Forbidden): pods is forbidden: User "jane.doe" cannot list resource "pods" in API group "" in the namespace "production"

Read this message literally — it tells you exactly which Role or ClusterRoleBinding you’re missing.

Summary

RBAC is the backbone of multi-tenant, secure Kubernetes operations. By combining Roles and ClusterRoles (what’s allowed) with RoleBindings and ClusterRoleBindings (who gets it), you can build precise, auditable access control for humans, CI pipelines, and controllers alike. Start from least privilege, use kubectl auth can-i liberally, and treat every wildcard rule as a red flag worth double-checking.

References

• Kubernetes Documentation: Using RBAC Authorization
• Kubernetes Documentation: Service Accounts
• CNCF Kubernetes Security Best Practices
• kubectl auth Reference

The post How to Use Role-Based Access Control (RBAC) in Kubernetes appeared first on Abdul Wahab Junaid.

Voluntary vs. Involuntary Disruptions

Kubernetes distinguishes between two kinds of Pod disruption:

• Involuntary disruptions: Hardware failure, kernel panic, node running out of resources — things you can’t prevent, only recover from (via replication).
• Voluntary disruptions: Things initiated by an administrator or automation — draining a node for maintenance, kubectl delete pod, cluster autoscaler scale-down, or a rolling node upgrade.

A PodDisruptionBudget only protects against voluntary disruptions. It tells Kubernetes: “No matter what maintenance operation you’re trying to perform, never take down more than X Pods (or leave fewer than Y available) from this set at once.”

Why This Matters for Loki Specifically

Loki’s architecture (especially in the microservices deployment mode) includes several stateful, quorum-sensitive components:

• Ingesters: Buffer and flush log data to long-term storage. Losing too many at once can cause data loss for logs not yet flushed.
• Distributors: Stateless, but if too many go down simultaneously, ingestion throughput craters.
• Queriers / Query Frontends: Handle read traffic; losing too many degrades query latency for everyone using Grafana dashboards.
• Compactor: Usually a single replica; losing it isn’t catastrophic short-term, but you don’t want it flapping constantly during node churn.

Because ingesters use a hash-ring based replication factor (commonly 3), losing more replicas than your replication factor tolerates during a rolling node upgrade can cause write failures or gaps in log ingestion. A PDB prevents the cluster from ever getting into that state during voluntary operations.

Kubernetes Architecture Refresher: Where PDBs Fit

PDBs are enforced by the Eviction API, which is what kubectl drain and the cluster autoscaler use instead of a raw delete. When something calls the Eviction API against a Pod:

1. The API server checks if any PDB covers that Pod (via label selector).
2. If evicting the Pod would violate the PDB’s minAvailable or maxUnavailable, the eviction is rejected with a 429 Too Many Requests.
3. The caller (drain, autoscaler) retries later, once the Pod count is back in a safe state.

Note: a PDB does not stop kubectl delete pod directly — it only governs the Eviction API. This distinction trips people up often.

Step 1: Deploy Loki with Helm

Assuming you’re using the official loki Helm chart in microservices mode:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki -n logging --create-namespace \
  --set loki.commonConfig.replication_factor=3 \
  --set deploymentMode=Distributed

Confirm the components are running:

kubectl get pods -n logging

You should see distributor, ingester, querier, query-frontend, and compactor Pods.

Step 2: Create a PDB for Loki Ingesters

Ingesters are the most disruption-sensitive component. With a replication factor of 3, you want to guarantee at least 2 are always available:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-ingester-pdb
  namespace: logging
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: ingester

Apply it:

kubectl apply -f loki-ingester-pdb.yaml
kubectl get pdb -n logging

Expected output:

NAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
loki-ingester-pdb  2               N/A                1                     10s

The ALLOWED DISRUPTIONS column tells you exactly how many Pods can be evicted right now without violating the budget — this is the number you should watch during a maintenance window.

Step 3: Create PDBs for Other Components

For distributors and queriers (stateless, but you still want availability during rolling maintenance), maxUnavailable is often more practical than minAvailable since it scales naturally with replica count:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-distributor-pdb
  namespace: logging
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: distributor
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: loki-querier-pdb
  namespace: logging
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: loki
      app.kubernetes.io/component: querier

Step 4: Configuring PDBs via the Helm Chart Values

Rather than managing PDBs as separate manifests, the Loki Helm chart supports PDB configuration natively — which is the cleaner, GitOps-friendly approach:

# values.yaml
ingester:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

querier:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

distributor:
  replicas: 3
  podDisruptionBudget:
    maxUnavailable: 1

helm upgrade loki grafana/loki -n logging -f values.yaml

Step 5: Test It — Simulate a Node Drain

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

If the drain would violate a PDB, you’ll see output like:

error when evicting pods/"loki-ingester-1" -n "logging": Cannot evict pod as it would violate the pod's disruption budget.

Kubernetes will keep retrying automatically as Pods on other nodes become healthy again, until it’s safe to proceed.

Monitoring Allowed Disruptions

For ongoing visibility, especially before a planned maintenance window, script a quick check:

kubectl get pdb -n logging -o custom-columns=\
NAME:.metadata.name,MIN_AVAIL:.spec.minAvailable,MAX_UNAVAIL:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed

If ALLOWED is 0 for any Loki component, that’s a signal something is already degraded — investigate before starting maintenance, not after.

Common Mistakes

• Setting minAvailable equal to total replica count — this makes the PDB impossible to satisfy during any voluntary disruption, effectively blocking node drains forever.
• Using label selectors that are too broad, accidentally covering Pods from other apps and creating confusing eviction blocks.
• Forgetting that PDBs don’t protect against involuntary disruptions (a node crashing outright bypasses the Eviction API entirely).
• Not aligning minAvailable/maxUnavailable with the actual replication factor configured in Loki’s commonConfig.replication_factor — the PDB should reflect the same fault tolerance the application itself expects.

High Availability and Disaster Recovery Considerations

• Spread Loki ingester replicas across multiple availability zones using pod anti-affinity or topology spread constraints, so a single zone failure doesn’t take out your entire replication factor at once — PDBs and anti-affinity work together, not as substitutes for each other.
• Back up Loki’s storage backend (S3, GCS, or equivalent) independently; PDBs protect availability during maintenance, not data durability.
• Combine PDBs with readiness probes — a Pod that’s technically “Running” but failing readiness checks doesn’t count as available, which affects how the PDB math is evaluated.

Summary

Pod Disruption Budgets are a small but critical piece of running Loki reliably on Kubernetes. By setting sensible minAvailable/maxUnavailable values per component — aligned with Loki’s own replication factor — you ensure that routine cluster maintenance never accidentally causes log ingestion failures or query outages. Configure them through the Helm chart for consistency, verify them with kubectl get pdb, and always test a drain in a non-production cluster before you rely on this in an actual maintenance window.

References

• Kubernetes Documentation: Pod Disruption Budgets
• Grafana Loki Documentation
• Grafana Loki Helm Chart
• CNCF Loki Project Page

The post How to Set Up Pod Disruption Budgets with Loki in Kubernetes appeared first on Abdul Wahab Junaid.

What Priority and Preemption Actually Do

Every Pod can carry a priority — an integer value derived from a PriorityClass object. When the scheduler can’t find a node with enough resources for a pending high-priority Pod, it may preempt (evict) lower-priority Pods on some node to make room, rather than leaving the high-priority Pod stuck in Pending indefinitely.

This is separate from and complementary to Pod Disruption Budgets — the scheduler respects PDBs where possible during preemption, but a PDB does not fully protect a Pod from being preempted if there’s no other way to schedule a higher-priority Pod.

Kubernetes Architecture: Where This Happens

Priority and preemption live entirely in the scheduler:

1. A Pod is submitted with a priorityClassName.
2. The admission controller resolves that name to a numeric priority value, stamped onto the Pod spec.
3. The scheduler tries to place it normally first.
4. If no node fits, and preemption is enabled (default), the scheduler looks for a node where evicting some lower-priority Pods would make room, chooses the node with the least “collateral damage,” and evicts just enough Pods.
5. The preempted Pods go back to Pending and get rescheduled elsewhere if capacity exists.

Step 1: Define PriorityClasses

PriorityClasses are cluster-scoped, so they’re usually created once, separately from your application charts (or as a shared “platform” chart installed first).

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical-priority
value: 1000000
globalDefault: false
description: "Reserved for critical production services."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: standard-priority
value: 100000
globalDefault: true
description: "Default priority for most application workloads."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-priority
value: 1000
globalDefault: false
preemptionPolicy: Never
description: "Low priority for batch and background jobs. Never preempts others."

kubectl apply -f priorityclasses.yaml
kubectl get priorityclass

Notice preemptionPolicy: Never on the batch class — this means Pods in that class will never trigger preemption of others, even though they still have a defined (low) priority. That’s a useful pattern for jobs that should wait patiently rather than kick anything else off a node.

Step 2: Reference PriorityClass in a Helm Chart

Rather than hardcoding priorityClassName in every Deployment template, expose it as a configurable value so different environments (or different releases of the same chart) can set it appropriately.

values.yaml:

priorityClassName: standard-priority

replicaCount: 3

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

templates/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-app
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}
    spec:
      {{- if .Values.priorityClassName }}
      priorityClassName: {{ .Values.priorityClassName }}
      {{- end }}
      containers:
        - name: app
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Step 3: Install with an Override per Environment

For a critical production release:

helm install payments-api ./chart -n production \
  --set priorityClassName=critical-priority

For a batch workload chart:

helm install nightly-report ./chart -n batch \
  --set priorityClassName=batch-priority

This gives you one chart, reused across workloads with wildly different scheduling importance, without duplicating templates.

Step 4: Verify Priority Assignment

kubectl get pod payments-api-xyz -o jsonpath='{.spec.priorityClassName}{"\n"}{.spec.priority}'

Expected output:

critical-priority
1000000

Observing Preemption in Action

When preemption occurs, check events on the newly scheduled Pod:

kubectl describe pod payments-api-xyz

You’ll see an event like:

Events:
  Type     Reason      Message
  ----     ------      -------
  Normal   Preempted   Preempted by payments-api-xyz on node worker-3

And on the evicted Pod’s side, it goes back to Pending, with a Preempting condition explaining why. The kube-scheduler logs (if you have access) also record the preemption decision in detail — useful for post-incident review of why a batch job got killed mid-run.

Handling Graceful Termination During Preemption

Preempted Pods still respect terminationGracePeriodSeconds, so a preempted Pod isn’t just SIGKILLed instantly — it gets its normal shutdown hooks. For workloads doing meaningful in-flight work, tune this appropriately in your chart:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60

Best Practices

• Don’t overuse high priority. If every team sets their workloads to the highest PriorityClass “just in case,” you lose the entire mechanism’s value — it becomes a race to the top rather than a meaningful signal.
• Reserve the top priority tier (e.g., system-cluster-critical, which Kubernetes itself uses for core components like kube-dns) for things that are genuinely cluster-critical; don’t let application teams use system-reserved classes.
• Combine with resource requests/limits and PodDisruptionBudgets — priority determines who gets preempted; PDBs limit how much can be preempted from a given set at once, giving you defense in depth.
• Set preemptionPolicy: Never on batch/best-effort classes so they never disrupt others, even though they can still be scheduled when capacity allows.
• Document your PriorityClass tiers in your platform’s onboarding docs so teams pick the correct one instead of guessing.

Common Mistakes

• Forgetting globalDefault: true must be set on exactly one PriorityClass — if none is marked default, unlabeled Pods get priority 0, which is lower than almost everything and can cause them to be preempted unexpectedly.
• Setting overly aggressive high-priority classes on non-critical workloads, causing cascading preemptions across the cluster during load spikes.
• Not testing preemption behavior in a staging cluster before rolling PriorityClasses into production — the first real preemption event shouldn’t be a surprise.

Summary

Pod Priority and Preemption give Kubernetes a principled way to decide who keeps running when resources get tight, and wiring priorityClassName into your Helm charts as a configurable value lets you apply consistent, environment-aware scheduling policy without duplicating manifests. Define a small, well-documented set of PriorityClasses, expose the choice through Helm values, and pair it with resource requests and PDBs for a cluster that degrades gracefully under pressure instead of falling over.

References

• Kubernetes Documentation: Pod Priority and Preemption
• Helm Documentation
• Kubernetes Scheduling Concepts
• CNCF Kubernetes Project

The post How to Implement Pod Priority and Preemption with Helm in Kubernetes appeared first on Abdul Wahab Junaid.

Why Splunk for Kubernetes

Splunk excels at correlating data across an entire organization — infrastructure, applications, security events — in one searchable index. If your security team already lives in Splunk for SIEM purposes, routing Kubernetes observability data there too means one pane of glass, one query language (SPL), and one retention/compliance policy, instead of maintaining Splunk alongside a separate Prometheus stack.

Kubernetes Architecture Context: What You’re Actually Monitoring

Before wiring up any tool, it helps to be clear on what data sources exist in a cluster:

• Container logs: stdout/stderr from every container, written to disk on each node by the container runtime.
• kubelet /metrics/cadvisor: Per-container CPU, memory, disk, and network usage.
• kube-state-metrics: Cluster-level object state (Deployment replica counts, Pod phase, node conditions) — this is not the same as resource usage; it’s object metadata as metrics.
• API server metrics: Request latency, error rates — critical for cluster health.
• Application metrics: Whatever your app exposes, often in Prometheus exposition format.

A complete monitoring setup pulls from all of these.

Step 1: Install the Splunk OpenTelemetry Collector via Helm

Splunk’s officially supported path for Kubernetes is the Splunk Distribution of the OpenTelemetry Collector, deployed as a Helm chart that runs both a node-level agent (DaemonSet) and a cluster-level receiver (Deployment).

helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
helm repo update

Create your values file:

# splunk-otel-values.yaml
splunkObservability:
  realm: us1
  accessToken: "<YOUR_SPLUNK_ACCESS_TOKEN>"

clusterName: production-cluster

logsEngine: otel

splunkPlatform:
  endpoint: "https://splunk-hec.yourcompany.com:8088/services/collector"
  token: "<YOUR_HEC_TOKEN>"
  index: "kubernetes"
  metricsIndex: "k8s_metrics"
  insecureSkipVerify: false

metricsEnabled: true
tracesEnabled: true
logsEnabled: true

Install it:

helm install splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector \
  -n monitoring --create-namespace \
  -f splunk-otel-values.yaml

Verify:

kubectl get pods -n monitoring

Expect to see an agent DaemonSet Pod on every node, plus a k8s-cluster-receiver Deployment Pod.

Step 2: Understand the Two Collector Roles

• Agent (DaemonSet): Runs on every node, tails container log files directly off disk, scrapes kubelet/cAdvisor metrics for that node’s Pods, and receives traces from local application SDKs.
• Cluster Receiver (single Deployment): Watches the Kubernetes API for cluster-wide object state (this is where kube-state-metrics-equivalent data comes from) so you’re not duplicating that collection on every node.

This split matters for resource planning — the agent’s resource requests scale with node count, while the cluster receiver stays a single low-overhead Pod regardless of cluster size.

Step 3: Verify Data Is Reaching Splunk

In Splunk Search:

index=kubernetes | stats count by k8s.namespace.name, k8s.pod.name

For metrics (if using Splunk Infrastructure Monitoring / Observability Cloud):

index=k8s_metrics metric_name="k8s.pod.cpu.utilization"
| stats avg(value) by k8s.pod.name

If nothing shows up after a few minutes, check the collector’s own logs first:

kubectl logs -n monitoring daemonset/splunk-otel-collector-agent

Look for HEC connection errors — the most common early failure is a wrong token or an unreachable HEC endpoint due to network policy or firewall rules.

Step 4: Configure HTTP Event Collector (HEC) on the Splunk Side

If you’re managing Splunk yourself (not Splunk Cloud), make sure HEC is enabled and a token exists with access to your target index:

# On the Splunk instance
splunk enable listen 8088 -auth admin:changeme

In Splunk Web: Settings → Data Inputs → HTTP Event Collector → New Token, scoped to the kubernetes and k8s_metrics indexes you created.

Step 5: Add Kubernetes Metadata Enrichment

By default, the collector enriches every log line and metric with Kubernetes metadata (namespace, pod name, labels, node) via the k8sattributes processor, which is already configured in the chart. You can extend it to pull specific custom labels your teams use for ownership tagging:

agent:
  config:
    processors:
      k8sattributes:
        extract:
          labels:
            - tag_name: team
              key: team
              from: pod
            - tag_name: environment
              key: env
              from: pod

This means every log and metric arriving in Splunk is automatically tagged with which team owns the Pod that generated it — critical for building dashboards and alerts scoped by team without manual tagging.

Step 6: Building Dashboards and Alerts

Once data is flowing, build a basic health dashboard in Splunk with panels like:

# Pod restart count in last hour
index=k8s_metrics metric_name="k8s.pod.restart_count"
| timechart span=5m max(value) by k8s.pod.name

# Error rate from application logs
index=kubernetes k8s.namespace.name="production" log_level="ERROR"
| timechart span=1m count

Set up alerts on conditions like sustained high restart counts, node NotReady status, or PVC usage crossing a threshold — all queryable the same way you’d alert on any other Splunk data source.

Security Considerations

• Use a dedicated Kubernetes ServiceAccount with RBAC scoped only to read (get, list, watch) on the resources the collector needs — never grant it write access.
• Store the Splunk access token and HEC token as Kubernetes Secrets, referenced via envFrom, never hardcoded in values files committed to Git.
• Enable TLS between the collector and your Splunk HEC endpoint (insecureSkipVerify: false in production, always).

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: splunk-otel-collector
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "namespaces", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets"]
    verbs: ["get", "list", "watch"]

Performance and Cost Optimization

• Filter noisy logs at the collector, not after ingestion — dropping health-check log spam before it hits Splunk saves significant licensing cost, since Splunk pricing is typically volume-based.
• Downsample high-cardinality metrics where per-second granularity isn’t needed.
• Use index-time field extraction sparingly — prefer search-time extraction where possible to reduce indexing overhead.

Common Mistakes

• Forgetting to set clusterName uniquely per cluster — without it, data from multiple clusters gets conflated in Splunk, making dashboards useless in multi-cluster environments.
• Not setting resource limits on the agent DaemonSet, letting it consume unbounded memory on log-heavy nodes.
• Sending 100% of logs from noisy sidecars (like service mesh proxies) without filtering, drowning out application signal.

High Availability

Run the cluster receiver Deployment with at least 2 replicas behind leader election (the chart supports this natively) so a single Pod restart doesn’t create a gap in cluster-level metric collection. The DaemonSet agent is inherently HA per-node since each node has its own instance.

Summary

Splunk can absolutely serve as your primary Kubernetes observability backend, and the Splunk OpenTelemetry Collector Helm chart is the well-supported way to get there — combining a DaemonSet agent for node-local logs/metrics/traces with a cluster receiver for object-level state. Get HEC configured correctly first, verify data with simple SPL queries, then layer in metadata enrichment, dashboards, and alerts once you trust the pipeline.

References

• Splunk OpenTelemetry Collector for Kubernetes
• Splunk OpenTelemetry Collector Helm Chart
• Kubernetes Documentation: Tools for Monitoring Resources
• CNCF OpenTelemetry Project

The post How to Set Up Kubernetes Monitoring with Splunk appeared first on Abdul Wahab Junaid.

The Core Problem It Solves

By default, if the scheduler can’t find room for a Pod, that Pod just sits in Pending forever, waiting for capacity to free up on its own. For most workloads, that’s fine. But for something genuinely important — an API gateway, a database primary — waiting indefinitely isn’t acceptable. Priority and Preemption lets Kubernetes actively make room for important Pods by evicting less important ones.

How Kubernetes Architecture Handles This

Two API objects and one scheduler behavior make this work:

1. PriorityClass (cluster-scoped): Maps a name to an integer priority value.
2. Pod.spec.priorityClassName: References a PriorityClass; the admission controller resolves it into Pod.spec.priority at creation time.
3. kube-scheduler: During scheduling, if a Pod can’t fit anywhere as-is, the scheduler runs its preemption logic — searching for a node where evicting some subset of lower-priority Pods would free enough resources, then evicts the minimum necessary set.

This all happens inside kube-scheduler; no other component is involved in the decision, though the API server and kubelet carry out the actual eviction and rescheduling.

Step 1: Understand Default Behavior

If you never create a PriorityClass, every Pod has priority 0 and preemption essentially never triggers in a meaningful way, because everything is equally “important” (or unimportant). Priority only becomes useful once you deliberately tier your workloads.

Check existing PriorityClasses, including built-in system ones:

kubectl get priorityclass

Typical output on any cluster:

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            30d
system-node-critical      2000001000   false            30d

These extremely high values are reserved for core cluster components like kube-dns and CNI plugins — application workloads should never use these classes.

Step 2: Create Custom PriorityClasses

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 100000
globalDefault: false
description: "High priority for customer-facing production services."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: default-priority
value: 10000
globalDefault: true
description: "Default for general application workloads."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 100
globalDefault: false
preemptionPolicy: Never
description: "Best-effort batch and background workloads."

kubectl apply -f priorityclasses.yaml

Step 3: Assign Priority to a Pod

apiVersion: v1
kind: Pod
metadata:
  name: important-api
spec:
  priorityClassName: high-priority
  containers:
    - name: api
      image: myrepo/api:latest
      resources:
        requests:
          cpu: "1"
          memory: 1Gi

kubectl apply -f important-api.yaml
kubectl get pod important-api -o jsonpath='{.spec.priority}'
# 100000

Step 4: Force a Preemption Scenario (for Learning)

To see preemption in action in a test cluster, fill a node close to capacity with low-priority Pods, then schedule a high-priority Pod requesting more than the currently free capacity:

kubectl apply -f low-priority-filler.yaml   # 5 replicas at low-priority
kubectl apply -f important-api.yaml         # high-priority, needs the space

Watch events:

kubectl get events --sort-by='.lastTimestamp' | grep -i preempt

You’ll see something like:

Normal  Preempted  pod/low-priority-filler-2  Preempted by important-api on node worker-1

The evicted Pod returns to Pending and gets rescheduled elsewhere once capacity exists — assuming your cluster has room; if not, it stays pending, which is expected.

PreemptionPolicy: Never

Setting preemptionPolicy: Never on a PriorityClass means Pods in that class will queue for available capacity like anything else, but will never cause other Pods to be evicted, even if their nominal priority is technically higher than some running Pod. This is the right choice for batch jobs that are “important but patient” — they shouldn’t disrupt live traffic just because someone gave them a high priority number for queue-ordering purposes.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: patient-batch
value: 50000
preemptionPolicy: Never

How the Scheduler Chooses What to Evict

The scheduler doesn’t just grab the lowest-priority Pod on any node — it tries to minimize collateral damage:

• It only considers Pods with strictly lower priority than the pending Pod.
• It prefers nodes where evicting the fewest Pods (or those causing least disruption) makes room.
• It respects PodDisruptionBudgets where possible, though a PDB is not an absolute guarantee against preemption if there’s truly no other option — preemption can still violate a PDB as a last resort, unlike voluntary eviction via kubectl drain, which strictly respects PDBs.
• Graceful termination still applies — evicted Pods get their normal terminationGracePeriodSeconds.

Interaction with Other Scheduling Features

• Node affinity / taints and tolerations: Preemption only considers nodes the pending Pod could actually run on given its affinity rules and tolerations — it won’t preempt Pods on a node the pending Pod couldn’t schedule to anyway.
• Pod Disruption Budgets: Provide a soft guarantee, not a hard block, against preemption.
• Cluster Autoscaler: If preemption alone can’t free enough room and the cluster can scale, the autoscaler may add nodes instead — preemption and autoscaling work as complementary mechanisms, not substitutes.

Best Practices

• Keep your PriorityClass tiers small and well-documented — 3 to 5 tiers is usually plenty (system, critical, standard, batch).
• Reserve very high values for platform/system workloads only.
• Use resource requests accurately — the scheduler’s preemption math is only as good as the requests Pods declare; underdeclared requests lead to unpredictable preemption behavior.
• Combine with ResourceQuotas per namespace so a single team can’t flood the cluster with high-priority Pods and starve everyone else.
• Test preemption behavior deliberately in staging before your first real production incident forces you to learn it live.

Common Mistakes

• Assigning high priority to workloads “just in case,” which inflates priority creep across the org until the mechanism becomes meaningless.
• Not setting a globalDefault PriorityClass, leaving unlabeled Pods at priority 0 — often lower than intended, making them the first candidates for preemption.
• Assuming PDBs fully protect against preemption; they don’t guarantee it during the “last resort” scheduling path.
• Ignoring preemptionPolicy: Never for batch workloads, causing unnecessary disruption of running work by lower-urgency jobs.

Troubleshooting

If a Pod stays Pending with FailedScheduling events even though lower-priority Pods exist:

kubectl describe pod important-api

Check the event reason carefully — it may be that no single node has enough evictable lower-priority capacity, even though the cluster in aggregate does. Preemption only considers one node at a time; it doesn’t combine partial evictions across multiple nodes for a single Pod.

Summary

Priority and Preemption gives Kubernetes a built-in way to enforce “important work wins” during resource contention, without requiring manual intervention. The mechanism is entirely scheduler-driven: PriorityClasses define the tiers, Pods reference them, and the scheduler evicts the minimum necessary lower-priority Pods to make room for higher-priority ones — always respecting graceful termination and, where possible, disruption budgets. Use it deliberately, with a small number of well-understood tiers, and it becomes one of the more powerful reliability tools in your cluster.

References

• Kubernetes Documentation: Pod Priority and Preemption
• Kubernetes Documentation: PriorityClass API Reference
• Kubernetes Scheduling Concepts Overview
• CNCF Kubernetes Project

The post How to Use Priority and Preemption in Kubernetes appeared first on Abdul Wahab Junaid.

Why Thanos Needs Careful PDB Planning

Thanos is composed of several distinct components, each with different failure characteristics:

• Sidecar: Runs alongside each Prometheus Pod, uploads blocks to object storage, and serves Prometheus’s local data to Queriers. Tightly coupled to its Prometheus instance.
• Querier: Stateless, fans out queries across Sidecars and Store Gateways, deduplicates results.
• Store Gateway: Serves historical data from object storage; often stateful with local caching.
• Compactor: Downsamples and compacts blocks in object storage — typically a single replica, since concurrent compaction against the same storage bucket can cause corruption.
• Receiver: Accepts remote-write traffic in HA setups; loses data if too many replicas go down before flushing.

Because the Compactor must run as a singleton, and the Receiver/Store Gateway have real availability requirements, a one-size-fits-all PDB strategy doesn’t work here — each component needs its own budget tuned to its actual replication and failure tolerance.

Kubernetes Architecture Refresher: How PDBs Are Enforced

A PDB doesn’t block direct kubectl delete pod calls. It only governs the Eviction API, which is what kubectl drain, the Cluster Autoscaler, and managed node upgrades use. When an eviction request comes in:

1. The API server checks all PDBs whose selector matches the target Pod.
2. If evicting would drop available replicas below minAvailable (or exceed maxUnavailable), the request is denied with 429 Too Many Requests.
3. The evicting controller retries later, once conditions allow.

This means PDBs protect you specifically during voluntary cluster operations — node drains, cordoning for upgrades, and autoscaler scale-downs — which is exactly when a multi-component system like Thanos is most at risk of losing more replicas at once than it can tolerate.

Step 1: Deploy Thanos via Helm

Using the community kube-prometheus-stack or standalone thanos chart:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install thanos bitnami/thanos -n monitoring --create-namespace \
  --set query.replicaCount=2 \
  --set storegateway.replicaCount=2 \
  --set receive.replicaCount=3 \
  --set compactor.enabled=true

Check what’s running:

kubectl get pods -n monitoring -l app.kubernetes.io/name=thanos

Step 2: PDB for the Querier (Stateless, Safe to Disrupt Gradually)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-query-pdb
  namespace: monitoring
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-query

With 2 replicas and maxUnavailable: 1, at least one Querier always stays up to serve Grafana dashboards during node maintenance.

Step 3: PDB for the Store Gateway

Store Gateways cache index data locally and can be expensive to “warm up” again after a restart, so it’s worth being conservative:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-storegateway-pdb
  namespace: monitoring
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-storegateway

With minAvailable: 1 on a 2-replica StatefulSet, only one can be evicted at a time — the other keeps serving historical query traffic while its sibling restarts and re-warms its cache.

Step 4: PDB for the Receiver (Data-Loss Sensitive)

Receivers accept remote-write traffic; taking down too many at once during a rolling node upgrade can cause dropped samples if your remote-write clients don’t buffer well. With 3 replicas and a typical replication factor of 2 in the Receiver hashring:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: thanos-receive-pdb
  namespace: monitoring
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-receive

This guarantees the hashring never drops below the minimum needed to satisfy its own replication factor during voluntary disruptions.

Step 5: The Compactor — Deliberately No PDB (or a Trivial One)

Since the Compactor typically runs as a single replica by design (running more than one against the same bucket risks corrupting compacted blocks), a minAvailable: 1 PDB would make it impossible to ever drain the node it’s on. For a singleton like this, it’s usually better to not create a PDB at all, and instead rely on it simply restarting elsewhere after eviction — accept the short gap in compaction rather than blocking cluster maintenance indefinitely.

# Deliberately omitted: no PDB for thanos-compactor
# Rationale: single replica by design; a minAvailable:1 PDB would
# permanently block node drains for the node it's scheduled on.

If you want some protection without blocking drains forever, use maxUnavailable: 0 combined with a PriorityClass and tight pod anti-affinity instead — but understand this still allows involuntary disruption, it just discourages voluntary ones without hard-blocking them indefinitely (this requires care and isn’t a universal recommendation).

Step 6: Apply and Verify

kubectl apply -f thanos-pdbs.yaml
kubectl get pdb -n monitoring

NAME                       MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
thanos-query-pdb           N/A             1                 1                     2m
thanos-storegateway-pdb    1               N/A               1                     2m
thanos-receive-pdb         2               N/A               1                     2m

Step 7: Simulate a Drain and Observe

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Watch how eviction proceeds component by component — Queriers and Store Gateways drain one at a time per their budgets, while Receiver Pods stop being evicted once only 2 remain, with the drain command retrying automatically until the third becomes safe to move (once a replacement is healthy elsewhere).

High Availability and Disaster Recovery

• Spread Thanos components across multiple availability zones with topology spread constraints — PDBs limit disruption during maintenance, but zone-level redundancy protects against an entire zone outage, which PDBs cannot do anything about.
• Ensure object storage (S3/GCS/Azure Blob) used by Thanos has its own durability guarantees and versioning — PDBs protect compute availability, not the underlying data.
• Regularly test full restores from the object storage bucket to confirm your Store Gateways and Compactor can rebuild state if the cluster itself is lost.

Common Mistakes

• Applying the same PDB template to every component without considering each one’s actual replication semantics — this is the single most common mistake with Thanos specifically, since it’s not architecturally uniform like a simple stateless web app.
• Forgetting the Compactor is a singleton and accidentally blocking node drains with an overly strict PDB.
• Setting minAvailable on the Receiver lower than its hashring replication factor, silently risking dropped writes during maintenance even though the PDB “passes.”

Summary

Thanos isn’t one workload — it’s several, each with distinct availability semantics, and your PDB strategy needs to reflect that. Give the Querier and Store Gateway conservative budgets, protect the Receiver hashring’s replication factor explicitly, and think carefully before applying any PDB at all to the singleton Compactor. Done right, this lets you drain and upgrade nodes confidently without silently degrading your monitoring stack’s own reliability — which is the last thing you want to fail quietly.

References

• Kubernetes Documentation: Pod Disruption Budgets
• Thanos Documentation
• Thanos GitHub Repository
• CNCF Thanos Project Page

The post How to Set Up Pod Disruption Budgets with Thanos in Kubernetes appeared first on Abdul Wahab Junaid.

Why StatefulSets Exist

A Deployment’s Pods are fungible — Pod names are randomly suffixed, storage is typically shared or ephemeral, and Pods can be created/destroyed in any order. That model breaks for anything that needs:

• Stable, predictable network identity (Pod-0 is always Pod-0, even after a restart).
• Stable storage per replica — Pod-0 always gets its own PersistentVolumeClaim back, not a random one.
• Ordered, graceful deployment and scaling — Pod-0 must be Running and Ready before Pod-1 starts, which matters for things like database replication bootstrapping.

A StatefulSet guarantees all three.

Kubernetes Architecture: What Makes a StatefulSet Different

• Stable identity: Pods are named <statefulset-name>-0, <statefulset-name>-1, etc., not randomly suffixed.
• Headless Service required: StatefulSets need a Service with clusterIP: None to provide stable DNS entries per Pod (pod-0.service-name.namespace.svc.cluster.local).
• volumeClaimTemplates: Instead of one shared PVC, each replica gets its own PVC, created from a template, and that PVC follows the same-named Pod across rescheduling.
• Ordered rolling updates by default: Updates happen in reverse ordinal order (highest number first) unless you configure podManagementPolicy: Parallel.

Step 1: Scaffold a Helm Chart

helm create my-statefulset-app
cd my-statefulset-app
rm templates/deployment.yaml templates/hpa.yaml templates/service.yaml

Step 2: Define the Headless Service

templates/headless-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: {{ include "my-statefulset-app.fullname" . }}-headless
  labels:
    {{- include "my-statefulset-app.labels" . | nindent 4 }}
spec:
  clusterIP: None
  selector:
    {{- include "my-statefulset-app.selectorLabels" . | nindent 4 }}
  ports:
    - name: app
      port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.targetPort }}

Step 3: Define the StatefulSet Template

templates/statefulset.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ include "my-statefulset-app.fullname" . }}
  labels:
    {{- include "my-statefulset-app.labels" . | nindent 4 }}
spec:
  serviceName: {{ include "my-statefulset-app.fullname" . }}-headless
  replicas: {{ .Values.replicaCount }}
  podManagementPolicy: {{ .Values.podManagementPolicy | default "OrderedReady" }}
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  selector:
    matchLabels:
      {{- include "my-statefulset-app.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "my-statefulset-app.selectorLabels" . | nindent 8 }}
    spec:
      terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds | default 30 }}
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: {{ .Values.service.targetPort }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          volumeMounts:
            - name: data
              mountPath: {{ .Values.persistence.mountPath }}
          readinessProbe:
            tcpSocket:
              port: {{ .Values.service.targetPort }}
            initialDelaySeconds: 10
            periodSeconds: 5
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ {{ .Values.persistence.accessMode | quote }} ]
        storageClassName: {{ .Values.persistence.storageClass }}
        resources:
          requests:
            storage: {{ .Values.persistence.size }}

Step 4: Values File

values.yaml:

replicaCount: 3

image:
  repository: myrepo/statefuldb
  tag: "1.0.0"

service:
  port: 5432
  targetPort: 5432

persistence:
  size: 10Gi
  accessMode: ReadWriteOnce
  storageClass: "fast-ssd"
  mountPath: /var/lib/data

podManagementPolicy: OrderedReady
terminationGracePeriodSeconds: 60

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    cpu: "1"
    memory: 2Gi

Step 5: Install and Verify

helm install mydb ./my-statefulset-app -n data --create-namespace
kubectl get statefulset -n data
kubectl get pods -n data -w

Expected Pod creation order:

mydb-0   0/1   Pending
mydb-0   1/1   Running
mydb-1   0/1   Pending
mydb-1   1/1   Running
mydb-2   0/1   Pending
mydb-2   1/1   Running

Each Pod only starts once the previous one is Running and Ready — this is the ordered guarantee at work.

Step 6: Verify Stable Storage and Identity

kubectl get pvc -n data

NAME           STATUS   VOLUME    CAPACITY   ACCESS MODES
data-mydb-0    Bound    pvc-abc   10Gi       RWO
data-mydb-1    Bound    pvc-def   10Gi       RWO
data-mydb-2    Bound    pvc-ghi   10Gi       RWO

Delete mydb-1 and watch it come back with the same PVC, not a fresh empty volume:

kubectl delete pod mydb-1 -n data
kubectl get pod mydb-1 -n data -o jsonpath='{.spec.volumes[0].persistentVolumeClaim.claimName}'
# data-mydb-1

Check DNS resolution from another Pod in the cluster:

kubectl run -it --rm debug --image=busybox -n data -- nslookup mydb-1.mydb-headless.data.svc.cluster.local

Rolling Updates with Partitions

For careful, staged rollouts (e.g., testing a new version on just the highest-ordinal replica first), use rollingUpdate.partition:

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    partition: 2

With partition: 2 on a 3-replica set, only mydb-2 gets updated on a chart upgrade; mydb-0 and mydb-1 stay untouched until you lower the partition value. This is a standard canary pattern for stateful workloads where you want to validate before wider rollout.

helm upgrade mydb ./my-statefulset-app --set image.tag=1.1.0 --set statefulset.partition=2
# validate mydb-2, then:
helm upgrade mydb ./my-statefulset-app --set image.tag=1.1.0 --set statefulset.partition=0

Scaling Considerations

Scaling up adds new ordinals sequentially (mydb-3 after mydb-0..2 are healthy); scaling down removes the highest ordinal first, not an arbitrary one. This matters for anything using replica index for identity, like a database where node 0 is always primary by convention.

kubectl scale statefulset mydb --replicas=5 -n data

Note that scaling down does not delete the PVCs by default — they’re retained so you don’t lose data by accident. Clean them up explicitly if you actually want the storage gone:

kubectl delete pvc data-mydb-4 data-mydb-3 -n data

Security and Best Practices

• Set podManagementPolicy: Parallel only for stateless-within-a-StatefulSet cases where ordering genuinely doesn’t matter (e.g., you just want stable identity/storage without startup ordering) — most real stateful apps need OrderedReady, the default.
• Always define a readiness probe that reflects actual application health (e.g., can accept queries, has joined a cluster), not just “process is running” — this is what the ordering guarantee depends on.
• Use PodDisruptionBudgets alongside StatefulSets to protect quorum-based systems (etcd, Kafka, Cassandra) from losing too many replicas during node maintenance.
• Set terminationGracePeriodSeconds high enough for graceful shutdown (e.g., a database flushing to disk) — the default 30s is often too short for real workloads.
• For disaster recovery, remember volumeClaimTemplates only protect against Pod rescheduling, not storage backend failure — pair with regular backups appropriate to your storage class/provider.

Common Mistakes

• Forgetting the headless Service — without it, per-Pod DNS simply won’t resolve, breaking peer discovery for clustered apps.
• Assuming scaling down deletes PVCs automatically — it doesn’t, which is a safety feature but surprises people expecting Deployment-like cleanup behavior.
• Using emptyDir instead of volumeClaimTemplates for state that needs to survive Pod rescheduling.
• Setting overly aggressive readiness probes that flap, causing the ordered rollout to stall waiting for a Pod that’s actually healthy but misreporting.

Summary

StatefulSets solve a real problem that Deployments can’t: stable identity, stable storage, and ordered lifecycle management for genuinely stateful workloads. Packaging one in Helm mainly means being deliberate about the headless Service, volumeClaimTemplates, and update/scaling behavior — none of which are optional details, since getting them wrong breaks the exact guarantees you chose a StatefulSet for in the first place.

References

• Kubernetes Documentation: StatefulSets
• Kubernetes Documentation: Headless Services
• Helm Documentation: Chart Template Guide
• CNCF Kubernetes Project

The post How to Implement StatefulSets with Helm in Kubernetes appeared first on Abdul Wahab Junaid.

Why Zabbix for Kubernetes

Zabbix gives you a single, mature monitoring platform across bare metal, VMs, network devices, and now Kubernetes — with built-in alerting, escalation policies, and a long history of enterprise reliability. If your infrastructure team already has Zabbix triggers, templates, and dashboards built out, extending that same system into Kubernetes avoids maintaining two separate alerting stacks with two separate on-call workflows.

Kubernetes Architecture Context

Zabbix’s Kubernetes integration works differently from the typical Prometheus pull model. It uses:

• Zabbix Agent 2, deployed as a DaemonSet, collecting node and container-level metrics.
• Zabbix Java Gateway or the newer native Kubernetes monitoring feature, which polls the Kubernetes API server directly for cluster-level object state (Pods, Deployments, Nodes) — conceptually similar to what kube-state-metrics does for Prometheus.
• Zabbix Server, the central component that stores collected data, evaluates triggers, and fires alerts.
• Zabbix Proxy (optional), useful in multi-cluster setups to reduce load on the central server and handle network segmentation.

Step 1: Deploy Zabbix Components via Helm

helm repo add zabbix-community https://zabbix-community.github.io/helm-zabbix
helm repo update

Install the Zabbix server, web frontend, and database:

helm install zabbix zabbix-community/zabbix -n zabbix --create-namespace \
  --set zabbixServer.enabled=true \
  --set zabbixWeb.enabled=true \
  --set mysql.enabled=true

Check the deployment:

kubectl get pods -n zabbix

Step 2: Deploy the Zabbix Agent as a DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: zabbix-agent
  namespace: zabbix
spec:
  selector:
    matchLabels:
      app: zabbix-agent
  template:
    metadata:
      labels:
        app: zabbix-agent
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: zabbix-agent2
          image: zabbix/zabbix-agent2:6.4-alpine-latest
          env:
            - name: ZBX_SERVER_HOST
              value: "zabbix-server.zabbix.svc.cluster.local"
            - name: ZBX_HOSTNAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: sys
              mountPath: /host/sys
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys

kubectl apply -f zabbix-agent-daemonset.yaml

Step 3: Configure Native Kubernetes Monitoring

Modern Zabbix (6.4+) includes a native Kubernetes monitoring data collection feature that polls the API server directly, rather than requiring a separate exporter. Configure it through the Zabbix frontend:

1. Data collection → Kubernetes → Create Kubernetes cluster
2. Provide the API server URL and a ServiceAccount token with read-only access (see RBAC below).
3. Zabbix auto-discovers Nodes, Pods, and Deployments and creates corresponding hosts and items automatically.

Create the ServiceAccount and token Zabbix will authenticate with:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: zabbix-monitoring
  namespace: zabbix
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: zabbix-monitoring-role
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "namespaces", "events", "services"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: zabbix-monitoring-binding
subjects:
  - kind: ServiceAccount
    name: zabbix-monitoring
    namespace: zabbix
roleRef:
  kind: ClusterRole
  name: zabbix-monitoring-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
  name: zabbix-monitoring-token
  namespace: zabbix
  annotations:
    kubernetes.io/service-account.name: zabbix-monitoring
type: kubernetes.io/service-account-token

Retrieve the token to paste into the Zabbix frontend:

kubectl get secret zabbix-monitoring-token -n zabbix -o jsonpath='{.data.token}' | base64 -d

Note this uses read-only verbs (get, list, watch) exclusively — Zabbix never needs write access to your cluster to monitor it.

Step 4: Install the Kubernetes Metrics Server (Required for Resource Metrics)

Zabbix’s metrics.k8s.io polling depends on the standard Kubernetes metrics-server being present in the cluster:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get deployment metrics-server -n kube-system

Without this, Zabbix can still see object state (Pod phase, restart counts) but won’t get live CPU/memory utilization numbers.

Step 5: Import the Kubernetes Nodes/Pods Template

Zabbix ships official templates (“Kubernetes nodes by HTTP”, “Kubernetes cluster state”) that map directly onto the discovered objects. Import via Data collection → Templates → Import, then link them to the auto-discovered host group for your cluster.

Step 6: Build Triggers and Alerts

Common triggers worth configuring immediately:

• Node NotReady condition sustained for more than 2 minutes.
• Pod CrashLoopBackOff state detected.
• Deployment’s available replicas below desired replicas for more than 5 minutes.
• PersistentVolumeClaim usage above 85%.

Example trigger expression (Zabbix trigger syntax) for a Pod restart spike:

last(/Kubernetes cluster state/kube.pod.restarts.rate[production,api-server])>5

Set escalation actions in Alerts → Actions → Trigger actions to route these into whatever notification channel your team already uses in Zabbix (email, Slack integration, PagerDuty via webhook).

Dashboards

Zabbix’s built-in dashboard widgets (Graph, Problems, Top hosts) can be composed into a Kubernetes overview dashboard without needing a separate tool like Grafana, though Zabbix also supports a Grafana data source plugin if your team prefers that visualization layer while keeping Zabbix as the collection/alerting backend.

Security Considerations

• Scope the monitoring ServiceAccount strictly to read-only verbs, as shown above — never grant create/update/delete.
• Run the Zabbix Agent DaemonSet with only the host mounts it actually needs (/proc, /sys read-only); avoid unnecessary hostPID/hostNetwork if your monitoring needs don’t require them.
• Store the Kubernetes API token as a Kubernetes Secret and restrict who can read it via RBAC on the zabbix namespace itself.
• If exposing the Zabbix web frontend externally, put it behind proper authentication (LDAP/SAML integration, which Zabbix supports natively) and TLS.

Performance and Scaling

• For large clusters (500+ nodes), consider a Zabbix Proxy per cluster or region to offload polling and reduce load on the central Zabbix Server, syncing data asynchronously.
• Tune the discovery interval for Kubernetes objects — very frequent polling on large clusters can generate significant load on both the API server and Zabbix’s own database.
• Archive/trend older historical data according to your retention needs; Zabbix’s housekeeping settings control this independently of Kubernetes.

Common Mistakes

• Forgetting to deploy metrics-server, then wondering why CPU/memory graphs are empty even though Pod state monitoring works fine.
• Granting the monitoring ServiceAccount write permissions “just in case” — unnecessary and a real security risk if the Zabbix server itself is ever compromised.
• Not setting resource limits on the Zabbix Agent DaemonSet, letting it compete with actual workloads for node resources.
• Over-polling large clusters without a Proxy tier, causing API server load spikes that show up as false “Node NotReady” alerts.

Summary

Zabbix can serve as a genuine, capable Kubernetes monitoring solution — not just a bolt-on — by combining its DaemonSet-based agent for node/container metrics with its native Kubernetes API polling for cluster object state. The setup takes a bit more manual RBAC and template work than a Prometheus stack, but it pays off if you’re consolidating Kubernetes into an observability platform your team already trusts and knows how to operate at scale.

References

• Zabbix Documentation: Kubernetes Monitoring
• Zabbix Helm Charts
• Kubernetes Metrics Server
• Kubernetes Documentation: Resource Metrics Pipeline

The post How to Set Up Kubernetes Monitoring with Zabbix appeared first on Abdul Wahab Junaid.