How to Use Priority and Preemption in Kubernetes

Every cluster eventually hits a moment where demand outpaces capacity — a traffic spike, a batch of jobs kicked off at the same time, or a node lost to maintenance. When that happens, Kubernetes needs a way to decide whose Pods get to run. That’s what Priority and Preemption is for. This guide covers the mechanism in depth, independent of any specific tool like Helm — how it works internally, how to configure it, and how to reason about its effects on scheduling.

The Core Problem It Solves

By default, if the scheduler can’t find room for a Pod, that Pod just sits in Pending forever, waiting for capacity to free up on its own. For most workloads, that’s fine. But for something genuinely important — an API gateway, a database primary — waiting indefinitely isn’t acceptable. Priority and Preemption lets Kubernetes actively make room for important Pods by evicting less important ones.

How Kubernetes Architecture Handles This

Two API objects and one scheduler behavior make this work:

1. PriorityClass (cluster-scoped): Maps a name to an integer priority value.
2. Pod.spec.priorityClassName: References a PriorityClass; the admission controller resolves it into Pod.spec.priority at creation time.
3. kube-scheduler: During scheduling, if a Pod can’t fit anywhere as-is, the scheduler runs its preemption logic — searching for a node where evicting some subset of lower-priority Pods would free enough resources, then evicts the minimum necessary set.

This all happens inside kube-scheduler; no other component is involved in the decision, though the API server and kubelet carry out the actual eviction and rescheduling.

Step 1: Understand Default Behavior

If you never create a PriorityClass, every Pod has priority 0 and preemption essentially never triggers in a meaningful way, because everything is equally “important” (or unimportant). Priority only becomes useful once you deliberately tier your workloads.

Check existing PriorityClasses, including built-in system ones:

kubectl get priorityclass

Typical output on any cluster:

NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-cluster-critical   2000000000   false            30d
system-node-critical      2000001000   false            30d

These extremely high values are reserved for core cluster components like kube-dns and CNI plugins — application workloads should never use these classes.

Step 2: Create Custom PriorityClasses

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 100000
globalDefault: false
description: "High priority for customer-facing production services."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: default-priority
value: 10000
globalDefault: true
description: "Default for general application workloads."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 100
globalDefault: false
preemptionPolicy: Never
description: "Best-effort batch and background workloads."

kubectl apply -f priorityclasses.yaml

Step 3: Assign Priority to a Pod

apiVersion: v1
kind: Pod
metadata:
  name: important-api
spec:
  priorityClassName: high-priority
  containers:
    - name: api
      image: myrepo/api:latest
      resources:
        requests:
          cpu: "1"
          memory: 1Gi

kubectl apply -f important-api.yaml
kubectl get pod important-api -o jsonpath='{.spec.priority}'
# 100000

Step 4: Force a Preemption Scenario (for Learning)

To see preemption in action in a test cluster, fill a node close to capacity with low-priority Pods, then schedule a high-priority Pod requesting more than the currently free capacity:

kubectl apply -f low-priority-filler.yaml   # 5 replicas at low-priority
kubectl apply -f important-api.yaml         # high-priority, needs the space

Watch events:

kubectl get events --sort-by='.lastTimestamp' | grep -i preempt

You’ll see something like:

Normal  Preempted  pod/low-priority-filler-2  Preempted by important-api on node worker-1

The evicted Pod returns to Pending and gets rescheduled elsewhere once capacity exists — assuming your cluster has room; if not, it stays pending, which is expected.

PreemptionPolicy: Never

Setting preemptionPolicy: Never on a PriorityClass means Pods in that class will queue for available capacity like anything else, but will never cause other Pods to be evicted, even if their nominal priority is technically higher than some running Pod. This is the right choice for batch jobs that are “important but patient” — they shouldn’t disrupt live traffic just because someone gave them a high priority number for queue-ordering purposes.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: patient-batch
value: 50000
preemptionPolicy: Never

How the Scheduler Chooses What to Evict

The scheduler doesn’t just grab the lowest-priority Pod on any node — it tries to minimize collateral damage:

• It only considers Pods with strictly lower priority than the pending Pod.
• It prefers nodes where evicting the fewest Pods (or those causing least disruption) makes room.
• It respects PodDisruptionBudgets where possible, though a PDB is not an absolute guarantee against preemption if there’s truly no other option — preemption can still violate a PDB as a last resort, unlike voluntary eviction via kubectl drain, which strictly respects PDBs.
• Graceful termination still applies — evicted Pods get their normal terminationGracePeriodSeconds.

Interaction with Other Scheduling Features

• Node affinity / taints and tolerations: Preemption only considers nodes the pending Pod could actually run on given its affinity rules and tolerations — it won’t preempt Pods on a node the pending Pod couldn’t schedule to anyway.
• Pod Disruption Budgets: Provide a soft guarantee, not a hard block, against preemption.
• Cluster Autoscaler: If preemption alone can’t free enough room and the cluster can scale, the autoscaler may add nodes instead — preemption and autoscaling work as complementary mechanisms, not substitutes.

Best Practices

• Keep your PriorityClass tiers small and well-documented — 3 to 5 tiers is usually plenty (system, critical, standard, batch).
• Reserve very high values for platform/system workloads only.
• Use resource requests accurately — the scheduler’s preemption math is only as good as the requests Pods declare; underdeclared requests lead to unpredictable preemption behavior.
• Combine with ResourceQuotas per namespace so a single team can’t flood the cluster with high-priority Pods and starve everyone else.
• Test preemption behavior deliberately in staging before your first real production incident forces you to learn it live.

Common Mistakes

• Assigning high priority to workloads “just in case,” which inflates priority creep across the org until the mechanism becomes meaningless.
• Not setting a globalDefault PriorityClass, leaving unlabeled Pods at priority 0 — often lower than intended, making them the first candidates for preemption.
• Assuming PDBs fully protect against preemption; they don’t guarantee it during the “last resort” scheduling path.
• Ignoring preemptionPolicy: Never for batch workloads, causing unnecessary disruption of running work by lower-urgency jobs.

Troubleshooting

If a Pod stays Pending with FailedScheduling events even though lower-priority Pods exist:

kubectl describe pod important-api

Check the event reason carefully — it may be that no single node has enough evictable lower-priority capacity, even though the cluster in aggregate does. Preemption only considers one node at a time; it doesn’t combine partial evictions across multiple nodes for a single Pod.

Summary

Priority and Preemption gives Kubernetes a built-in way to enforce “important work wins” during resource contention, without requiring manual intervention. The mechanism is entirely scheduler-driven: PriorityClasses define the tiers, Pods reference them, and the scheduler evicts the minimum necessary lower-priority Pods to make room for higher-priority ones — always respecting graceful termination and, where possible, disruption budgets. Use it deliberately, with a small number of well-understood tiers, and it becomes one of the more powerful reliability tools in your cluster.

References

• Kubernetes Documentation: Pod Priority and Preemption
• Kubernetes Documentation: PriorityClass API Reference
• Kubernetes Scheduling Concepts Overview
• CNCF Kubernetes Project