How to Implement Node Affinity in Kubernetes

If you’ve ever run a mixed workload cluster — some nodes with GPUs, some in a specific availability zone, some with local SSDs — you’ve probably run into the problem of pods landing on the wrong hardware. I hit this exact issue a few years back when a batch of machine learning pods kept getting scheduled onto nodes without GPUs, and the jobs just sat there crash-looping. That’s the moment I actually sat down and learned node affinity properly instead of just copy-pasting nodeSelector snippets from Stack Overflow.

This guide walks through node affinity from the ground up: what it is, how the scheduler actually uses it, and how to configure it correctly in production.

What Is Node Affinity?

Node affinity is a Kubernetes scheduling feature that lets you constrain which nodes your pod is eligible to be scheduled on, based on labels on the node. It’s the more expressive successor to nodeSelector — think of nodeSelector as the blunt instrument and node affinity as the scalpel.

Node affinity lets you express rules like:

• “This pod must run on a node with SSD storage.”
• “This pod should preferably run in zone us-east-1a, but it’s not a hard requirement.”
• “Never schedule this pod on a node labeled spot=true.”

Where Node Affinity Fits in Kubernetes Architecture

To understand why node affinity works the way it does, it helps to understand the scheduling pipeline:

1. A pod is created and lands in the kube-apiserver‘s pending queue.
2. The kube-scheduler watches for unscheduled pods.
3. The scheduler runs a filtering phase — eliminating nodes that don’t satisfy hard constraints (resource requests, taints, required affinity rules).
4. It then runs a scoring phase — ranking the remaining nodes based on preferences (including preferred affinity rules).
5. The highest-scoring node is selected, and a binding is created.

Node affinity rules are evaluated in both phases depending on whether they’re “required” (filtering) or “preferred” (scoring).

Node Affinity Types

There are two types of node affinity, and understanding the difference is the whole game:

1. requiredDuringSchedulingIgnoredDuringExecution

This is a hard requirement. If no node matches, the pod stays Pending. The “IgnoredDuringExecution” part means that if node labels change after the pod is already running, the pod is not evicted — the rule is only checked at scheduling time.

2. preferredDuringSchedulingIgnoredDuringExecution

This is a soft preference. The scheduler tries to honor it but will still schedule the pod elsewhere if no matching node is available.

There’s a requiredDuringSchedulingRequiredDuringExecution type planned in the API for future eviction-on-label-change behavior, but as of current stable Kubernetes releases, it isn’t implemented — don’t rely on it.

Labeling Your Nodes

Before you can use affinity, your nodes need labels. Kubernetes auto-applies some (like topology.kubernetes.io/zone), but you’ll often add your own.

kubectl label nodes worker-node-1 disktype=ssd
kubectl label nodes worker-node-2 disktype=hdd
kubectl label nodes worker-node-3 gpu=true

Verify:

kubectl get nodes --show-labels

Output:

NAME            STATUS   ROLES    AGE   VERSION   LABELS
worker-node-1   Ready    <none>   10d   v1.30.2   disktype=ssd,kubernetes.io/hostname=worker-node-1
worker-node-2   Ready    <none>   10d   v1.30.2   disktype=hdd,kubernetes.io/hostname=worker-node-2
worker-node-3   Ready    <none>   10d   v1.30.2   gpu=true,kubernetes.io/hostname=worker-node-3

Basic Example: Required Node Affinity

Here’s a full pod manifest requiring SSD-backed nodes:

apiVersion: v1
kind: Pod
metadata:
  name: ssd-required-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
  containers:
  - name: app
    image: nginx:1.27
    resources:
      requests:
        cpu: "250m"
        memory: "256Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"

Apply and check:

kubectl apply -f ssd-required-pod.yaml
kubectl get pod ssd-required-pod -o wide

If no node has disktype=ssd, you’ll see:

kubectl describe pod ssd-required-pod

Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector.

This is your first troubleshooting checkpoint — that exact message means your affinity rule filtered out every node.

Preferred Node Affinity Example

Now let’s say GPU nodes are preferred but not mandatory — useful when GPU capacity is limited and you’d rather run somewhere than not run at all:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: inference-service
  template:
    metadata:
      labels:
        app: inference-service
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: gpu
                operator: In
                values:
                - "true"
          - weight: 20
            preference:
              matchExpressions:
              - key: disktype
                operator: In
                values:
                - ssd
      containers:
      - name: inference
        image: myregistry.io/inference-service:2.3.1
        resources:
          requests:
            cpu: "1"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "4Gi"

The weight field (1–100) controls how strongly each preference influences the scoring phase. Higher weight, stronger pull toward matching nodes.

Combining Multiple Match Expressions

Match expressions within a single nodeSelectorTerms entry are ANDed together; multiple nodeSelectorTerms entries are ORed. This trips people up constantly, so here’s a concrete example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values: ["ssd"]
        - key: zone
          operator: In
          values: ["us-east-1a"]
      - matchExpressions:
        - key: instance-type
          operator: In
          values: ["m5.xlarge"]

This means: schedule on a node that is (SSD AND zone-1a) OR (instance-type m5.xlarge).

Supported operators: In, NotIn, Exists, DoesNotExist, Gt, Lt.

Node Affinity vs Taints and Tolerations vs Pod Affinity

These three mechanisms are often confused:

• Node affinity: pod says which nodes it wants.
• Taints/tolerations: node says which pods it will accept (a node repels pods unless they tolerate the taint).
• Pod affinity/anti-affinity: pod says which other pods it wants to be near or away from.

They’re complementary, not competing. A common production pattern combines a taint (to keep general workloads off GPU nodes) with node affinity (so GPU workloads actively seek those nodes) and a matching toleration.

# On the GPU pod
spec:
  tolerations:
  - key: "gpu-only"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: In
            values: ["true"]

Production Use Case: Multi-AZ High Availability

A very common real-world pattern is spreading replicas across availability zones using node affinity combined with topology labels:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-frontend
spec:
  replicas: 6
  selector:
    matchLabels:
      app: web-frontend
  template:
    metadata:
      labels:
        app: web-frontend
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-east-1a
                - us-east-1b
                - us-east-1c
      containers:
      - name: web
        image: myregistry.io/web-frontend:4.1.0
        ports:
        - containerPort: 8080

Note this only restricts which zones are eligible — it doesn’t guarantee even spread across them. For that you need Pod Topology Spread Constraints (a separate mechanism worth pairing with this).

CI/CD Integration

In a GitOps workflow, node affinity rules typically live in Helm values so different environments can target different node pools:

# values-production.yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: environment
          operator: In
          values: ["production"]

# templates/deployment.yaml
spec:
  template:
    spec:
      affinity:
        {{- toYaml .Values.affinity | nindent 8 }}

This keeps affinity rules environment-specific without duplicating manifest logic, and it plugs cleanly into a pipeline that runs helm upgrade --install per environment.

Node Affinity with Cluster Autoscaler

When node affinity rules restrict pods to a subset of nodes that don’t currently exist (e.g., a GPU node pool scaled to zero to save cost), the Cluster Autoscaler needs to understand your affinity requirements to scale up the correct node group rather than leaving the pod stuck Pending indefinitely. This works automatically as long as your node groups are labeled to match what your affinity rules expect:

# Example: GKE node pool with matching labels for autoscaler simulation
gcloud container node-pools create gpu-pool \
  --cluster=my-cluster \
  --node-labels=gpu=true \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=5

The autoscaler simulates scheduling for pending pods against each configured node group’s template (including its labels), determines which group would satisfy the pod’s affinity rules, and scales that specific group — this is why keeping node pool labels and affinity rules in sync matters more than it might initially seem; a mismatch here silently prevents scale-up entirely, with the pod just sitting Pending and no clear error explaining why the autoscaler didn’t act.

Node Affinity vs Pod Affinity: A Quick Distinction

It’s worth being precise about terminology here since the names are easy to conflate. Everything covered in this article is node affinity — rules about node labels. Pod affinity/anti-affinity is a related but distinct mechanism where a pod’s placement depends on the labels of other pods already running, not node labels directly:

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: cache
      topologyKey: kubernetes.io/hostname

This example would require the pod to be scheduled on a node that already has a pod labeled app: cache running on it — useful for co-locating a service with its cache to reduce network latency. Node affinity and pod affinity are frequently combined in the same pod spec when both hardware requirements and pod co-location matter simultaneously.

Common Mistakes

• Using requiredDuringSchedulingIgnoredDuringExecution everywhere. This turns every label typo into a stuck Pending pod. Reserve “required” for genuinely hard constraints.
• Forgetting labels drift. If someone relabels a node and forgets to update the affinity rule, pods silently stop landing where expected.
• Confusing AND/OR semantics between matchExpressions and nodeSelectorTerms, as shown above.
• Not combining with taints. Affinity alone doesn’t repel other workloads from your specialized nodes — you need taints for that.
• No fallback plan. If you require GPU nodes and your GPU node pool scales to zero, pods just queue up. Consider preferred affinity plus autoscaler node pools for elasticity.

Troubleshooting Checklist

# Check why a pod is pending
kubectl describe pod <pod-name>

# Confirm node labels match what you expect
kubectl get nodes --show-labels

# Check scheduler logs for detailed filtering decisions
kubectl logs -n kube-system -l component=kube-scheduler

# Dry-run to validate YAML syntax
kubectl apply -f pod.yaml --dry-run=client -o yaml

Performance Considerations

Node affinity evaluation adds minor overhead to scheduling, but it’s negligible compared to gains from correct placement — for example avoiding cross-zone network costs, or making sure latency-sensitive pods land on nodes with local NVMe storage. In large clusters (1000+ nodes), heavily nested matchExpressions across many pods can slow scheduling throughput slightly; keep expressions as simple as your use case allows.

Summary

Node affinity gives you fine-grained control over pod placement using label-based rules, with both hard (required) and soft (preferred) variants. It’s foundational for running heterogeneous clusters — mixed hardware, multi-AZ deployments, and specialized workloads like GPU inference. Pair it with taints/tolerations for full control over specialized node pools, and with topology spread constraints for even distribution. Start with preferred affinity where possible; reserve required affinity for constraints that genuinely cannot be violated.

References

• Kubernetes Documentation: Assigning Pods to Nodes
• Kubernetes Scheduler Documentation
• Kubernetes API Reference: NodeAffinity
• CNCF Kubernetes Project