How to Use DaemonSets in Kubernetes

If you’ve spent any time running Kubernetes in production, you’ve probably run into a situation where you needed a specific pod to run on every single node in your cluster — not three replicas, not five, but exactly one per node, no matter how many nodes you have or how often that number changes. That’s precisely the problem DaemonSets were built to solve, and in this guide I’m going to walk you through everything from the basic concept to advanced production patterns on AWS EKS.

I’ll cover what a DaemonSet actually is under the hood, how the scheduler treats it differently from a Deployment, how to write and deploy real manifests, how to update and roll back DaemonSets safely, and how to monitor and troubleshoot them when things go sideways.

What Is a DaemonSet?

A DaemonSet is a Kubernetes controller that ensures a copy of a specific pod runs on all (or a selected subset of) nodes in a cluster. As nodes are added to the cluster, the DaemonSet controller automatically schedules the pod onto them. As nodes are removed, those pods are garbage collected. This is fundamentally different from a Deployment, which cares about maintaining a desired replica count regardless of which nodes those replicas land on.

Common real-world use cases include:

• Log collection agents — Fluentd, Fluent Bit, or Filebeat shipping logs from every node
• Monitoring agents — Node Exporter for Prometheus, Datadog agent, New Relic infrastructure agent
• Networking components — CNI plugins like Calico or Cilium, kube-proxy itself is often run as a DaemonSet
• Storage daemons — Ceph or GlusterFS storage daemons that need direct node access
• Security agents — Falco, Aqua Security, or other node-level security scanners

How DaemonSets Work Internally

Understanding the internals helps you reason about failure modes later. The DaemonSet controller watches the cluster’s node list via the API server. For every node that matches the DaemonSet’s node selector (or all nodes, if none is specified), it creates a pod spec bound to that node using .spec.nodeName, which bypasses the normal scheduler filtering for resource fit in older Kubernetes versions.

Since Kubernetes 1.12, DaemonSet pods are scheduled by the default scheduler rather than the DaemonSet controller directly, using a NodeAffinity term. This matters because it means DaemonSet pods respect the same taints, tolerations, and node affinity rules as any other pod, but they’re specifically designed to tolerate the taints that would normally repel workloads — like node-role.kubernetes.io/control-plane:NoSchedule — so that monitoring and logging daemons can still run on control-plane nodes when needed.

Each DaemonSet pod bypasses normal kube-scheduler load-balancing logic in the sense that it doesn’t compete for “best fit” placement — it’s guaranteed one-per-matching-node regardless of resource pressure, unless you configure resource requests that the node can’t satisfy, in which case the pod will be stuck in Pending.

Basic DaemonSet Manifest

Let’s start with a straightforward example: deploying Fluent Bit as a log-shipping DaemonSet.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      terminationGracePeriodSeconds: 10
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

A few things worth noting here:

• hostPath volumes are extremely common in DaemonSets because the whole point is often to access node-level resources (logs, metrics, sockets like /var/run/docker.sock).
• The tolerations block is critical if you want the daemon to also run on control-plane/master nodes.
• There’s no replicas field — DaemonSets don’t use it, since the “replica count” is implicitly the number of matching nodes.

Apply it with:

kubectl apply -f fluent-bit-daemonset.yaml

Verify it landed on every node:

kubectl get daemonset fluent-bit -n logging

Expected output:

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluent-bit   3         3         3       3            3           <none>          2m

DESIRED here equals the number of nodes eligible to run the pod, and if CURRENT or READY lag behind, that’s your first troubleshooting signal.

Targeting a Subset of Nodes

Sometimes you don’t want a daemon on every node — for example, a GPU monitoring agent should only run on GPU-enabled nodes. Use nodeSelector or, better, affinity for more expressive rules.

spec:
  template:
    spec:
      nodeSelector:
        workload-type: gpu

Or with node affinity for more complex logic:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node.kubernetes.io/instance-type
                    operator: In
                    values:
                      - p3.2xlarge
                      - p3.8xlarge

On AWS EKS, this pairs naturally with managed node groups that use instance-type-specific labels, or with Karpenter provisioners tagged by workload type.

Update Strategies

DaemonSets support two update strategies, configured under .spec.updateStrategy.type:

• RollingUpdate (default since 1.6) — old pods are killed and new ones created gradually, respecting maxUnavailable.
• OnDelete — pods are only updated when you manually delete them, giving you full manual control.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0

maxSurge is only meaningful when it’s non-zero and lets new pods come up before old ones are terminated — useful for daemons where even brief downtime (like a CNI or service mesh sidecar-injector daemon) is disruptive to the whole node.

To trigger a rolling update, just change the pod spec (e.g., bump the image tag) and reapply:

kubectl set image daemonset/fluent-bit fluent-bit=fluent/fluent-bit:2.2.1 -n logging
kubectl rollout status daemonset/fluent-bit -n logging

To roll back:

kubectl rollout undo daemonset/fluent-bit -n logging

DaemonSets and Resource Management on EKS

Because a DaemonSet pod runs on every node, its resource requests are effectively a tax on every node’s allocatable capacity. This is one of the most common production mistakes: teams add several DaemonSets (log agent, monitoring agent, service mesh agent, security agent) without tracking cumulative resource requests, then wonder why regular workload pods won’t schedule due to insufficient CPU/memory.

Best practice: keep DaemonSet resource requests as low and precise as possible, and regularly audit total DaemonSet overhead per node:

kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.metadata.ownerReferences[]?.kind=="DaemonSet") | 
  "\(.metadata.namespace)/\(.metadata.name): cpu=\(.spec.containers[0].resources.requests.cpu // "none") mem=\(.spec.containers[0].resources.requests.memory // "none")"'

On EKS specifically, if you’re using Fargate profiles for certain namespaces, be aware that DaemonSets cannot run on Fargate — Fargate pods don’t have the underlying node access a DaemonSet needs, so any namespace fully on Fargate simply won’t get DaemonSet coverage. Plan your logging/monitoring architecture accordingly (e.g., use Fargate’s built-in log router instead).

Helm-Based Deployment

In production, most teams don’t hand-write DaemonSet YAML for common agents — they use Helm charts maintained by the project or vendor. For example, deploying the official Prometheus Node Exporter:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install node-exporter prometheus-community/prometheus-node-exporter \
  --namespace monitoring --create-namespace \
  --set resources.requests.cpu=50m \
  --set resources.requests.memory=30Mi

This gives you sane defaults, tolerations already configured for control-plane nodes, and upgrade paths managed via helm upgrade.

Monitoring DaemonSet Health

Beyond kubectl get daemonset, you’ll want ongoing visibility. A few practical commands:

# Check per-node pod status
kubectl get pods -n logging -o wide -l app=fluent-bit

# Describe to see scheduling failures
kubectl describe daemonset fluent-bit -n logging

# Check events for a stuck pod
kubectl get events -n logging --field-selector involvedObject.name=fluent-bit-abcde

If you’re running Prometheus, scrape kube_daemonset_status_desired_number_scheduled and kube_daemonset_status_number_ready from kube-state-metrics, and alert when desired != ready for more than a few minutes — that’s a strong signal of a stuck rollout or node-level scheduling failure (often insufficient resources or a taint mismatch).

Common Mistakes

• Forgetting tolerations for control-plane nodes when you actually need coverage there (e.g., control-plane log shipping).
• Not setting resource limits, which lets a misbehaving daemon (like a log agent hitting a backpressure loop) consume unbounded node resources and destabilize every workload on that node.
• Using hostNetwork: true without understanding port conflicts — since DaemonSet pods often need host networking, make sure the ports they bind don’t collide with other host-network daemons.
• Ignoring maxUnavailable during upgrades on small clusters — updating a 3-node cluster’s CNI DaemonSet with maxUnavailable: 1 still means a third of your cluster loses networking capability briefly during rollout; plan maintenance windows accordingly.

Disaster Recovery Considerations

DaemonSets are declarative, so recovery is largely about GitOps discipline: keep manifests in version control (or Helm values files), and treat cluster rebuilds as “reapply from source.” If you’re using tools like ArgoCD or Flux, DaemonSets sync automatically like any other resource. The main DR risk specific to DaemonSets is silent drift — a manually kubectl edit-ed DaemonSet that no longer matches source control — so enforce sync policies that revert manual drift.

Summary

DaemonSets solve a specific but common problem: guaranteeing exactly one instance of a pod per node for cluster-wide concerns like logging, monitoring, networking, and security. Unlike Deployments, they’re node-count-driven rather than replica-count-driven, they integrate with the standard scheduler via node affinity since 1.12, and they support both rolling and manual update strategies. On EKS, remember that DaemonSets don’t run on Fargate, and always budget resource requests carefully since every DaemonSet pod is a tax on every node’s capacity.

References

• Kubernetes Documentation: DaemonSet
• Kubernetes Documentation: Taints and Tolerations
• AWS EKS Documentation: Fargate Considerations
• Prometheus Community Helm Charts
• CNCF Fluent Bit Documentation