How to Use Taints and Tolerations in Kubernetes

The first time I really understood taints and tolerations was after a bad afternoon where a batch analytics job kept getting scheduled onto the same nodes as latency-sensitive API pods, tanking response times during a run. Node affinity alone couldn’t fix it because the API pods didn’t know they needed to avoid anything — they had no rule pushing them away. That’s the gap taints and tolerations exist to close.

The Core Idea

Node affinity is about pods choosing nodes. Taints and tolerations are the inverse: nodes repelling pods. A taint is applied to a node and says “don’t schedule anything here unless it explicitly tolerates this.” A toleration on a pod says “I’m allowed to ignore this particular taint.”

This distinction matters a lot in practice. Affinity is opt-in from the pod’s side; taints are opt-out enforced from the node’s side. You typically want taints when you need a strong guarantee that only intended workloads land somewhere — dedicated GPU nodes, spot instances, or nodes reserved for a specific team.

How Scheduling With Taints Works

During the filtering phase of scheduling, kube-scheduler checks every node’s taints against the pod’s tolerations. If a node has a taint the pod doesn’t tolerate, that node is eliminated from consideration — full stop, no scoring involved for that node.

Taint Structure

A taint has three parts: key, value, and effect.

kubectl taint nodes worker-node-3 dedicated=gpu:NoSchedule

This reads as: key=dedicated, value=gpu, effect=NoSchedule.

The Three Effects

• NoSchedule — the scheduler will not place new pods on this node unless they tolerate the taint. Existing pods are unaffected.
• PreferNoSchedule — a soft version; the scheduler tries to avoid the node but will use it if necessary.
• NoExecute — the strongest effect. New pods without a matching toleration won’t be scheduled, and existing pods without the toleration are evicted.

Applying Taints

# Standard taint
kubectl taint nodes worker-node-3 dedicated=gpu:NoSchedule

# Verify
kubectl describe node worker-node-3 | grep Taints

Output:

Taints:             dedicated=gpu:NoSchedule

Remove a taint by appending a -:

kubectl taint nodes worker-node-3 dedicated=gpu:NoSchedule-

Writing Tolerations

A toleration on a pod must match the taint’s key, value, and effect (or use Exists to match any value).

apiVersion: v1
kind: Pod
metadata:
  name: gpu-training-job
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  containers:
  - name: trainer
    image: myregistry.io/ml-trainer:1.4.0
    resources:
      requests:
        cpu: "4"
        memory: "16Gi"
        nvidia.com/gpu: "1"
      limits:
        cpu: "8"
        memory: "32Gi"
        nvidia.com/gpu: "1"

Note: a toleration alone does not guarantee the pod lands on the tainted node — it only permits it. Pair it with node affinity to actively pull the pod there.

spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values:
            - gpu

This combo — taint to repel everyone else, affinity to pull in the intended workload, toleration to permit it — is the standard production pattern for dedicated node pools.

Using Exists Operator

To tolerate a taint regardless of its value:

tolerations:
- key: "dedicated"
  operator: "Exists"
  effect: "NoSchedule"

To tolerate all taints with a given effect, regardless of key:

tolerations:
- operator: "Exists"
  effect: "NoSchedule"

Use this sparingly — it’s a broad exemption and easy to misuse.

NoExecute and Eviction Timing

NoExecute taints can include tolerationSeconds, letting a pod stay for a grace period before eviction rather than being kicked out instantly:

tolerations:
- key: "node.kubernetes.io/unreachable"
  operator: "Exists"
  effect: "NoExecute"
  tolerationSeconds: 300

This is exactly how Kubernetes handles node failures internally — when a node becomes NotReady or Unreachable, the control plane automatically taints it, and pods without a toleration are evicted after the default grace period (typically 300 seconds), letting the ReplicaSet controller reschedule them elsewhere.

Built-In Taints You’ll Encounter

Kubernetes applies several taints automatically:

• node.kubernetes.io/not-ready — node controller detects the node isn’t ready.
• node.kubernetes.io/unreachable — node controller can’t reach the node.
• node.kubernetes.io/memory-pressure
• node.kubernetes.io/disk-pressure
• node.kubernetes.io/pid-pressure
• node.kubernetes.io/network-unavailable
• node.kubernetes.io/unschedulable — applied when you run kubectl cordon.

Control plane nodes are also tainted by default (node-role.kubernetes.io/control-plane:NoSchedule) so regular workloads don’t land there.

kubectl describe node <control-plane-node> | grep Taints

Taints:             node-role.kubernetes.io/control-plane:NoSchedule

Real-World Use Case: Spot/Preemptible Instance Pools

A very common cost-optimization pattern is running batch/stateless workloads on spot instances while keeping critical services on stable on-demand nodes.

kubectl taint nodes spot-node-1 lifecycle=spot:NoSchedule
kubectl taint nodes spot-node-2 lifecycle=spot:NoSchedule

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-worker
spec:
  replicas: 10
  selector:
    matchLabels:
      app: batch-worker
  template:
    metadata:
      labels:
        app: batch-worker
    spec:
      tolerations:
      - key: "lifecycle"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: lifecycle
                operator: In
                values:
                - spot
      containers:
      - name: worker
        image: myregistry.io/batch-worker:3.2.0

Using preferredDuringScheduling here rather than required means these pods will still schedule on regular nodes if the spot pool is temporarily unavailable — good practice for workloads that need to run but don’t strictly need spot pricing.

Helm Chart Pattern

Exposing taints/tolerations as configurable values keeps charts portable across environments:

# values.yaml
tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

# templates/deployment.yaml
spec:
  template:
    spec:
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

Troubleshooting

# Check why a pod won't schedule
kubectl describe pod <pod-name>

Look for:

Warning  FailedScheduling  default-scheduler  0/5 nodes are available: 3 node(s) had untolerated taint {dedicated: gpu}, 2 Insufficient cpu.

That message tells you exactly how many nodes were eliminated by taints versus other constraints — read it carefully before assuming affinity is the problem.

# List all taints across the cluster
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'

Taints and Tolerations in Cluster Autoscaler Node Groups

When using the Kubernetes Cluster Autoscaler with dedicated node pools (GPU, spot, or otherwise specialized), the autoscaler needs to understand which node group corresponds to which taint so it can scale up the correct pool when a pod with a matching toleration is stuck Pending. This is typically expressed via node group labels/taints configured at the cloud-provider level, mirrored in the autoscaler’s config:

# Example: AWS node group tags used by cluster autoscaler auto-discovery
# k8s.io/cluster-autoscaler/node-template/taint/dedicated: gpu:NoSchedule

aws autoscaling create-or-update-tags --tags \
  ResourceId=my-gpu-asg,ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/taint/dedicated,Value=gpu:NoSchedule,PropagateAtLaunch=true

Without this tag, the autoscaler doesn’t know the GPU node group’s taint ahead of time — it can still scale it up reactively once a pod is Pending, but pre-registering the taint lets it make correct simulated scheduling decisions faster, especially when choosing between multiple candidate node groups for a batch of pending pods.

Applying Taints via Node Pool Configuration (Cloud-Native Approach)

Rather than manually running kubectl taint after node creation — which doesn’t survive node replacement during upgrades or scaling events — apply taints directly at the node pool/group level so every node launched into that pool is tainted automatically from boot:

# EKS managed node group with a taint applied at creation
eksctl create nodegroup \
  --cluster multi-az-prod \
  --name gpu-workers \
  --node-type p3.2xlarge \
  --nodes 2 \
  --node-labels "gpu=true" \
  --node-taints "dedicated=gpu:NoSchedule"

This is the production-correct pattern — manually applied kubectl taint commands are fine for testing, but for anything long-lived, the taint should be defined as part of the node pool’s launch configuration so it’s self-healing across node replacements, not something a human has to remember to reapply.

Common Mistakes

• Assuming toleration = placement. A toleration only removes a barrier; it doesn’t attract the pod. Always pair with affinity when you need guaranteed placement.
• Using NoExecute without tolerationSeconds for workloads that should get a grace period during transient node issues — this can cause unnecessary churn.
• Tainting nodes but forgetting DaemonSets. DaemonSets that need to run everywhere (like log collectors or CNI agents) need explicit tolerations for every taint in the cluster, including NoSchedule on control-plane nodes.
• Overusing blanket Exists tolerations, which defeats the purpose of isolation.
• Not accounting for taints in cluster autoscaler configuration — the autoscaler needs to know which node groups satisfy which tolerations to scale the right pool.

Security Implications

Taints and tolerations aren’t a security boundary by themselves — a toleration is just YAML any pod author can add if they know (or guess) the taint’s key/value. For genuine isolation (e.g., regulatory workload segregation), combine taints with:

• Kubernetes RBAC limiting who can create pods with specific tolerations, potentially enforced via admission control.
• Pod Security Admission or an OPA/Gatekeeper policy restricting toleration usage by namespace.
• Dedicated node pools with restricted IAM/network access at the cloud provider level.

Summary

Taints repel pods from nodes; tolerations let specific pods override that repulsion. NoSchedule blocks new scheduling, PreferNoSchedule is a soft version, and NoExecute evicts existing pods too. The standard production pattern pairs taints (repel) with node affinity (attract) and tolerations (permit) to build dedicated node pools — for GPUs, spot instances, or team-specific hardware — without accidentally starving other workloads of capacity or letting the wrong pods slip in.

References

• Kubernetes Documentation: Taints and Tolerations
• Kubernetes Documentation: Node-pressure Eviction
• Kubernetes API Reference
• CNCF Kubernetes Project